Skip to contentSkip to conversation

Add enlightenments for vGPU

PATCH Superseded Yu Zhang 7 participants 58 messages Nov 13, 2014 – Dec 18, 2014

Subsystems (3)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Yu Zhang

author
#0 lore raw

v3 cover Add enlightenments for vGPU

Intel GVT-g (previously known as XenGT), is a complete GPU virtualization solution with mediated pass-through for 4th generation Intel Core processors - Haswell platform. This technology presents a virtual full-fledged GPU to each Virtual Machine (VM). VMs can directly access performance-critical resources, without intervention from the hypervisor in most cases, while privileged operations from VMs are trap-and-emulated at minimal cost. For detail, please refer to
https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-release

This patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, will be sent out in another patchset.

The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory w/o hypervisor's intervention, e.g. filling textures or queuing commands. However w/ the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses w/o complexity of translation.

Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (w/ some faulting mechanism).

The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource XenGT also partitions them among VMs.

Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.

Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vGPU in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vGPU in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vGPU specific write MMIO to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enabled

 Documentation/DocBook/drm.tmpl      |   5 +
 drivers/gpu/drm/i915/Makefile       |   3 +
 drivers/gpu/drm/i915/i915_dma.c     |  11 ++
 drivers/gpu/drm/i915/i915_drv.h     |  11 ++
 drivers/gpu/drm/i915/i915_gem.c     |   5 +
 drivers/gpu/drm/i915/i915_gem_gtt.c |  33 ++++-
 drivers/gpu/drm/i915/i915_vgpu.c    | 234 ++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_vgpu.h    |  94 +++++++++++++++
 drivers/gpu/drm/i915/intel_pm.c     |   8 ++
 drivers/gpu/drm/i915/intel_uncore.c |  19 +++
 10 files changed, 420 insertions(+), 3 deletions(-)
 create mode 100644 drivers/gpu/drm/i915/i915_vgpu.c
 create mode 100644 drivers/gpu/drm/i915/i915_vgpu.h
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#1 lore raw

PATCH v3 1/8 drm/i915: Introduce a PV INFO page structure for Intel GVT-g.

Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through. This page contains the shared information between i915 driver and the host emulator. For now, this structure utilizes an area of 4K bypes on HSW GPU's unused MMIO space to support existing production. Future hardware will have the reserved window architecturally defined, and layout of the page will be added in future BSpec.

The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. Thereafter a flag, vgpu.active is set, and intel_vgpu_active() is used by checking this flag to conclude if GPU is virtualized with Intel GVT-g. By now, intel_vgpu_active() will return true, only when the driver is running as a guest in the Intel GVT-g enhanced environment on HSW platform.

v2:
take Chris' comments:
- call the i915_check_vgpu() in intel_uncore_init()
- sanitize i915_check_vgpu() by adding BUILD_BUG_ON() and debug info
take Daniel's comments:
- put the definition of PV INFO into a new header - i915_vgt_if.h
other changes:
- access mmio regs by readq/readw in i915_check_vgpu()

v3:
take Daniel's comments:
- move the i915/vgt interfaces into a new i915_vgpu.c
- update makefile
- add kerneldoc to functions which are non-static
- add a DOC: section describing some of the high-level design
- update drm docbook
other changes:
- rename i915_vgt_if.h to i915_vgpu.h

---
 Documentation/DocBook/drm.tmpl      |  5 +++
 drivers/gpu/drm/i915/Makefile       |  3 ++
 drivers/gpu/drm/i915/i915_drv.h     | 11 +++++
 drivers/gpu/drm/i915/i915_vgpu.c    | 85 +++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_vgpu.h    | 85 +++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/intel_uncore.c |  3 ++
 6 files changed, 192 insertions(+)
 create mode 100644 drivers/gpu/drm/i915/i915_vgpu.c
 create mode 100644 drivers/gpu/drm/i915/i915_vgpu.h
Documentation/DocBook/drm.tmpl +5 -0
index 64d9c1e..e4db4cf 100644
@@ -3832,6 +3832,11 @@ int num_ioctls;</synopsis>
 !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_disable_interrupts
 !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_enable_interrupts
       </sect2>
+      <sect2>
+        <title>Intel GVT-g Guest Support(vGPU)</title>
+!Pdrivers/gpu/drm/i915/i915_vgpu.c Intel GVT-g guest support
+!Idrivers/gpu/drm/i915/i915_vgpu.c
+      </sect2>
     </sect1>
     <sect1>
       <title>Display Hardware Handling</title>
drivers/gpu/drm/i915/Makefile +3 -0
index 891e584..d9aa5ca 100644
@@ -79,6 +79,9 @@ i915-y += dvo_ch7017.o \
 	  intel_sdvo.o \
 	  intel_tv.o
 
+# virtual gpu code
+i915-y += i915_vgpu.o
+
 # legacy horrors
 i915-y += i915_dma.o \
 	  i915_ums.o
drivers/gpu/drm/i915/i915_drv.h +11 -0
index 54691bc..e1e221e 100644
@@ -1470,6 +1470,10 @@ struct i915_workarounds {
 	u32 count;
 };
 
+struct i915_virtual_gpu {
+	bool active;
+};
+
 struct drm_i915_private {
 	struct drm_device *dev;
 	struct kmem_cache *slab;
@@ -1482,6 +1486,8 @@ struct drm_i915_private {
 
 	struct intel_uncore uncore;
 
+	struct i915_virtual_gpu vgpu;
+
 	struct intel_gmbus gmbus[GMBUS_NUM_PORTS];
 
 
@@ -2311,6 +2317,11 @@ extern void intel_uncore_check_errors(struct drm_device *dev);
 extern void intel_uncore_fini(struct drm_device *dev);
 extern void intel_uncore_forcewake_reset(struct drm_device *dev, bool restore);
 
+static inline bool intel_vgpu_active(struct drm_device *dev)
+{
+	return to_i915(dev)->vgpu.active;
+}
+
 void
 i915_enable_pipestat(struct drm_i915_private *dev_priv, enum pipe pipe,
 		     u32 status_mask);
drivers/gpu/drm/i915/i915_vgpu.c new file +85 -0

85 lines not on this page · raw message

drivers/gpu/drm/i915/i915_vgpu.h new file +85 -0

85 lines not on this page · raw message

drivers/gpu/drm/i915/intel_uncore.c +3 -0
index 9427641..cae27bb 100644
@@ -23,6 +23,7 @@
 
 #include "i915_drv.h"
 #include "intel_drv.h"
+#include "i915_vgpu.h"
 
 #define FORCEWAKE_ACK_TIMEOUT_MS 2
 
@@ -850,6 +851,8 @@ void intel_uncore_init(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
+	i915_check_vgpu(dev);
+
 	setup_timer(&dev_priv->uncore.force_wake_timer,
 		    gen6_force_wake_timer, (unsigned long)dev_priv);
 
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#2 lore raw

Re: [PATCH v3 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.

Quoted 5 lines · Yu Zhang

Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through. This page contains the shared information between i915 driver and the host emulator. For now, this structure utilizes an area of 4K bypes on HSW GPU's unused MMIO space to support

bytes

Quoted 1 line

existing production. Future hardware will have the reserved window

It is slightly unclear to me what "existing production" means?

Quoted 148 lines

architecturally defined, and layout of the page will be added in future BSpec.

The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. Thereafter a flag, vgpu.active is set, and intel_vgpu_active() is used by checking this flag to conclude if GPU is virtualized with Intel GVT-g. By now, intel_vgpu_active() will return true, only when the driver is running as a guest in the Intel GVT-g enhanced environment on HSW platform.

v2:
take Chris' comments:
- call the i915_check_vgpu() in intel_uncore_init()
- sanitize i915_check_vgpu() by adding BUILD_BUG_ON() and debug info
take Daniel's comments:
- put the definition of PV INFO into a new header - i915_vgt_if.h
other changes:
- access mmio regs by readq/readw in i915_check_vgpu()

v3:
take Daniel's comments:
- move the i915/vgt interfaces into a new i915_vgpu.c
- update makefile
- add kerneldoc to functions which are non-static
- add a DOC: section describing some of the high-level design
- update drm docbook
other changes:
- rename i915_vgt_if.h to i915_vgpu.h

---
  Documentation/DocBook/drm.tmpl      |  5 +++
  drivers/gpu/drm/i915/Makefile       |  3 ++
  drivers/gpu/drm/i915/i915_drv.h     | 11 +++++
  drivers/gpu/drm/i915/i915_vgpu.c    | 85 +++++++++++++++++++++++++++++++++++++
  drivers/gpu/drm/i915/i915_vgpu.h    | 85 +++++++++++++++++++++++++++++++++++++
  drivers/gpu/drm/i915/intel_uncore.c |  3 ++
  6 files changed, 192 insertions(+)

create mode 100644 drivers/gpu/drm/i915/i915_vgpu.c create mode 100644 drivers/gpu/drm/i915/i915_vgpu.h

Documentation/DocBook/drm.tmpl +5 -0
index 64d9c1e..e4db4cf 100644
@@ -3832,6 +3832,11 @@ int num_ioctls;</synopsis>
  !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_disable_interrupts
  !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_enable_interrupts
        </sect2>
+      <sect2>
+        <title>Intel GVT-g Guest Support(vGPU)</title>
+!Pdrivers/gpu/drm/i915/i915_vgpu.c Intel GVT-g guest support
+!Idrivers/gpu/drm/i915/i915_vgpu.c
+      </sect2>
      </sect1>
      <sect1>
        <title>Display Hardware Handling</title>
drivers/gpu/drm/i915/Makefile +3 -0
index 891e584..d9aa5ca 100644
@@ -79,6 +79,9 @@ i915-y += dvo_ch7017.o \
  	  intel_sdvo.o \
  	  intel_tv.o
 
+# virtual gpu code
+i915-y += i915_vgpu.o
+
  # legacy horrors
  i915-y += i915_dma.o \
  	  i915_ums.o
drivers/gpu/drm/i915/i915_drv.h +11 -0
index 54691bc..e1e221e 100644
@@ -1470,6 +1470,10 @@ struct i915_workarounds {
  	u32 count;
  };
 
+struct i915_virtual_gpu {
+	bool active;
+};
+
  struct drm_i915_private {
  	struct drm_device *dev;
  	struct kmem_cache *slab;
@@ -1482,6 +1486,8 @@ struct drm_i915_private {
 
  	struct intel_uncore uncore;
 
+	struct i915_virtual_gpu vgpu;
+
  	struct intel_gmbus gmbus[GMBUS_NUM_PORTS];
 
 
@@ -2311,6 +2317,11 @@ extern void intel_uncore_check_errors(struct drm_device *dev);
  extern void intel_uncore_fini(struct drm_device *dev);
  extern void intel_uncore_forcewake_reset(struct drm_device *dev, bool restore);
 
+static inline bool intel_vgpu_active(struct drm_device *dev)
+{
+	return to_i915(dev)->vgpu.active;
+}
+
  void
  i915_enable_pipestat(struct drm_i915_private *dev_priv, enum pipe pipe,
  		     u32 status_mask);
drivers/gpu/drm/i915/i915_vgpu.c new file +32 -0
new file mode 100644
index 0000000..3f6b797
@@ -0,0 +1,85 @@
+/*
+ * Copyright(c) 2011-2014 Intel Corporation. All rights reserved.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include "intel_drv.h"
+#include "i915_vgpu.h"
+
+
+/**
+ * DOC: Intel GVT-g guest support
+ *
+ * Intel GVT-g is a graphics virtualization technology which shares the
+ * GPU among multiple virtual machines on a time-sharing basis. Each

drivers/gpu/drm/i915/i915_vgpu.c line 42

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +10 -0
+ * virtual machine is presented a virtual GPU (vGPU), which has equivalent
+ * features as the underlying physical GPU (pGPU), so i915 driver can run
+ * seamlessly in a virtual machine. This file provides vGPU specific
+ * optimizations when running in a virtual machine, to reduce the complexity
+ * of vGPU emulation and to improve the overall performance.
+ *
+ * A primary function introduced here is so-called "address space ballooning"
+ * technique. Intel GVT-g partitions global graphics memory among multiple VMs,
+ * so each VM can directly access a portion of the memory w/o hypervisor's
+ * intervention, e.g. filling textures or queuing commands. However w/ the

Personal preference, but It may be nicer to avoid "w/o" and similar shorthand in documentation.

Quoted 29 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +29 -0
+ * partitioning an unmodified i915 driver would assume a smaller graphics
+ * memory starting from address ZERO, then requires vGPU emulation module to
+ * translate the graphics address between 'guest view' and 'host view', for
+ * all registers and command opcodes which contain a graphics memory address.
+ * To reduce the complexity, Intel GVT-g introduces "address space ballooning",
+ * by telling the exact partitioning knowledge to each guest i915 driver, which
+ * then reserves and prevents non-allocated portions from allocation. Thus vGPU
+ * emulation module module only needs to scan and validate graphics addresses
+ * w/o complexity of address translation.
+ *
+ */
+
+/**
+ * i915_check_vgpu - detect virtual GPU
+ * @dev: drm device *
+ *
+ * This function is called at the initialization stage, to detect whether
+ * running on a vGPU.
+ */
+void i915_check_vgpu(struct drm_device *dev)
+{
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	uint64_t magic;
+	uint32_t version;
+
+	BUILD_BUG_ON(sizeof(struct vgt_if) != VGT_PVINFO_SIZE);
+
+	if (!IS_HASWELL(dev))
+		return;

drivers/gpu/drm/i915/i915_vgpu.c line 81

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +10 -0
+
+	magic = readq(dev_priv->regs + vgtif_reg(magic));
+	if (magic != VGT_MAGIC)
+		return;
+
+	version = INTEL_VGT_IF_VERSION_ENCODE(
+		readw(dev_priv->regs + vgtif_reg(version_major)),
+		readw(dev_priv->regs + vgtif_reg(version_minor)));
+	if (version != INTEL_VGT_IF_VERSION)
+		return;

Would it be useful to something on version mismatch?

Quoted 47 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +3 -0
+	dev_priv->vgpu.active = true;
+	DRM_INFO("Virtual GPU for Intel GVT-g detected.\n");
+}
drivers/gpu/drm/i915/i915_vgpu.h new file +38 -0
new file mode 100644
index 0000000..5f41d01c
@@ -0,0 +1,85 @@
+/*
+ * Copyright(c) 2011-2014 Intel Corporation. All rights reserved.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#ifndef _I915_VGPU_H_
+#define _I915_VGPU_H_
+
+/* The MMIO offset of the shared info between guest and host emulator */
+#define VGT_PVINFO_PAGE	0x78000
+#define VGT_PVINFO_SIZE	0x1000
+
+/*
+ * The following structure pages are defined in GEN MMIO space
+ * for virtualization. (One page for now)
+ */
+#define VGT_MAGIC         0x4776544776544776	/* 'vGTvGTvG' */
+#define VGT_VERSION_MAJOR 1
+#define VGT_VERSION_MINOR 0
+

drivers/gpu/drm/i915/i915_vgpu.h line 48

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +10 -0
+#define INTEL_VGT_IF_VERSION_ENCODE(major, minor) ((major) << 16 | (minor))
+#define INTEL_VGT_IF_VERSION \
+	INTEL_VGT_IF_VERSION_ENCODE(VGT_VERSION_MAJOR, VGT_VERSION_MINOR)
+
+struct vgt_if {
+	uint64_t magic;		/* VGT_MAGIC */
+	uint16_t version_major;
+	uint16_t version_minor;
+	uint32_t vgt_id;	/* ID of vGT instance */
+	uint32_t rsv1[12];	/* pad to offset 0x40 */

In general I thought stdint types should be avoided in the kernel in favour of u32 and company. But I see i915 is full of them already so maybe my information is outdated.

Quoted 11 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +11 -0
+	/*
+	 *  Data structure to describe the balooning info of resources.
+	 *  Each VM can only have one portion of continuous area for now.
+	 *  (May support scattered resource in future)
+	 *  (starting from offset 0x40)
+	 */
+	struct {
+		/* Aperture register balooning */
+		struct {
+			uint32_t my_base;
+			uint32_t my_size;

drivers/gpu/drm/i915/i915_vgpu.h line 69

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +10 -0
+		} low_gmadr;	/* aperture */
+		/* GMADR register balooning */
+		struct {
+			uint32_t my_base;
+			uint32_t my_size;
+		} high_gmadr;	/* non aperture */
+		/* allowed fence registers */
+		uint32_t fence_num;
+		uint32_t rsv2[3];
+	} avail_rs;		/* available/assigned resource */

I didn't figure out yet why the concept is to pass in two ranges to mark as reserved, instead of one usable/allowed range and then implicitly all the rest as reserved?

drivers/gpu/drm/i915/i915_vgpu.h line 78

Quoted 9 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +9 -0
+	uint32_t rsv3[0x200 - 24];	/* pad to half page */
+	/*
+	 * The bottom half page is for response from Gfx driver to hypervisor.
+	 * Set to reserved fields temporarily by now.
+	 */
+	uint32_t rsv4;
+	uint32_t display_ready;	/* ready for display owner switch */
+	uint32_t rsv5[0x200 - 2];	/* pad to one page */
+} __packed;

You probably don't need __packed since you have BUILD_BUG_ON on size mismatch and pad to full size with reserved fields. I don't think it matters though.

Quoted 20 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +6 -0
+#define vgtif_reg(x) \
+	(VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)
+
+extern void i915_check_vgpu(struct drm_device *dev);
+
+#endif /* _I915_VGPU_H_ */
drivers/gpu/drm/i915/intel_uncore.c +1 -0
index 9427641..cae27bb 100644
@@ -23,6 +23,7 @@
 
  #include "i915_drv.h"
  #include "intel_drv.h"
+#include "i915_vgpu.h"
 
  #define FORCEWAKE_ACK_TIMEOUT_MS 2
 

drivers/gpu/drm/i915/intel_uncore.c void intel_uncore_init(struct drm_device *dev) · line 858

Quoted 8 lines
drivers/gpu/drm/i915/intel_uncore.c +2 -0
@@ -850,6 +851,8 @@ void intel_uncore_init(struct drm_device *dev)
  {
  	struct drm_i915_private *dev_priv = dev->dev_private;
 
+	i915_check_vgpu(dev);
+
  	setup_timer(&dev_priv->uncore.force_wake_timer,
  		    gen6_force_wake_timer, (unsigned long)dev_priv);
 

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#3 lore raw

Re: [PATCH v3 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.

Quoted 10 lines · Tvrtko Ursulin

drivers/gpu/drm/i915/i915_vgpu.h line 48

Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +6 -0
+struct vgt_if {
+	uint64_t magic;		/* VGT_MAGIC */
+	uint16_t version_major;
+	uint16_t version_minor;
+	uint32_t vgt_id;	/* ID of vGT instance */
+	uint32_t rsv1[12];	/* pad to offset 0x40 */

In general I thought stdint types should be avoided in the kernel in favour of u32 and company. But I see i915 is full of them already so maybe my information is outdated.

That was back when c99 types weren't a common thing, nowadays I think we should prefer _t types for more uniformity with userspace. Also it makes the __u32 and similar types which are required for ioctl structs stand out much more. And yes we're totally inconsistent about it.

Just an aside.
-Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#4 lore raw

Re: [PATCH v3 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.

In reply to Tvrtko Ursulin

Thank you very much for your thorough review, Tvrtko. :)

Quoted 9 lines · Tvrtko Ursulin
Quoted 5 lines · Yu Zhang

Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through. This page contains the shared information between i915 driver and the host emulator. For now, this structure utilizes an area of 4K bypes on HSW GPU's unused MMIO space to support

bytes

Got it. Thanks.

Quoted 4 lines
Quoted 2 lines

existing production. Future hardware will have the reserved window

It is slightly unclear to me what "existing production" means?

Our current gpu virtualization implementation for HSW. It's compared with future implematations for BDW etc.

Quoted 181 lines
Quoted 181 lines

architecturally defined, and layout of the page will be added in future BSpec.

The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. Thereafter a flag, vgpu.active is set, and intel_vgpu_active() is used by checking this flag to conclude if GPU is virtualized with Intel GVT-g. By now, intel_vgpu_active() will return true, only when the driver is running as a guest in the Intel GVT-g enhanced environment on HSW platform.

v2:
take Chris' comments:
- call the i915_check_vgpu() in intel_uncore_init()
- sanitize i915_check_vgpu() by adding BUILD_BUG_ON() and debug info
take Daniel's comments:
- put the definition of PV INFO into a new header - i915_vgt_if.h
other changes:
- access mmio regs by readq/readw in i915_check_vgpu()

v3:
take Daniel's comments:
- move the i915/vgt interfaces into a new i915_vgpu.c
- update makefile
- add kerneldoc to functions which are non-static
- add a DOC: section describing some of the high-level design
- update drm docbook
other changes:
- rename i915_vgt_if.h to i915_vgpu.h

---
  Documentation/DocBook/drm.tmpl      |  5 +++
  drivers/gpu/drm/i915/Makefile       |  3 ++
  drivers/gpu/drm/i915/i915_drv.h     | 11 +++++
  drivers/gpu/drm/i915/i915_vgpu.c    | 85
+++++++++++++++++++++++++++++++++++++
  drivers/gpu/drm/i915/i915_vgpu.h    | 85
+++++++++++++++++++++++++++++++++++++
  drivers/gpu/drm/i915/intel_uncore.c |  3 ++
  6 files changed, 192 insertions(+)

create mode 100644 drivers/gpu/drm/i915/i915_vgpu.c create mode 100644 drivers/gpu/drm/i915/i915_vgpu.h

Documentation/DocBook/drm.tmpl +5 -0
index 64d9c1e..e4db4cf 100644
@@ -3832,6 +3832,11 @@ int num_ioctls;</synopsis>
  !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_disable_interrupts
  !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_enable_interrupts
        </sect2>
+      <sect2>
+        <title>Intel GVT-g Guest Support(vGPU)</title>
+!Pdrivers/gpu/drm/i915/i915_vgpu.c Intel GVT-g guest support
+!Idrivers/gpu/drm/i915/i915_vgpu.c
+      </sect2>
      </sect1>
      <sect1>
        <title>Display Hardware Handling</title>
drivers/gpu/drm/i915/Makefile +3 -0
index 891e584..d9aa5ca 100644
@@ -79,6 +79,9 @@ i915-y += dvo_ch7017.o \
        intel_sdvo.o \
        intel_tv.o
 
+# virtual gpu code
+i915-y += i915_vgpu.o
+
  # legacy horrors
  i915-y += i915_dma.o \
        i915_ums.o
drivers/gpu/drm/i915/i915_drv.h +11 -0
index 54691bc..e1e221e 100644
@@ -1470,6 +1470,10 @@ struct i915_workarounds {
      u32 count;
  };
 
+struct i915_virtual_gpu {
+    bool active;
+};
+
  struct drm_i915_private {
      struct drm_device *dev;
      struct kmem_cache *slab;
@@ -1482,6 +1486,8 @@ struct drm_i915_private {
 
      struct intel_uncore uncore;
 
+    struct i915_virtual_gpu vgpu;
+
      struct intel_gmbus gmbus[GMBUS_NUM_PORTS];
 
 
@@ -2311,6 +2317,11 @@ extern void intel_uncore_check_errors(struct
 drm_device *dev);
  extern void intel_uncore_fini(struct drm_device *dev);
  extern void intel_uncore_forcewake_reset(struct drm_device *dev,
 bool restore);
 
+static inline bool intel_vgpu_active(struct drm_device *dev)
+{
+    return to_i915(dev)->vgpu.active;
+}
+
  void

i915_enable_pipestat(struct drm_i915_private *dev_priv, enum pipe pipe,

               u32 status_mask);
drivers/gpu/drm/i915/i915_vgpu.c new file +4 -0
new file mode 100644
index 0000000..3f6b797
@@ -0,0 +1,85 @@
+/*
+ * Copyright(c) 2011-2014 Intel Corporation. All rights reserved.
+ *
+ * Permission is hereby granted, free of charge, to any person

obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */

drivers/gpu/drm/i915/i915_vgpu.c new file +11 -0
+
+#include "intel_drv.h"
+#include "i915_vgpu.h"
+
+
+/**
+ * DOC: Intel GVT-g guest support
+ *
+ * Intel GVT-g is a graphics virtualization technology which shares the
+ * GPU among multiple virtual machines on a time-sharing basis. Each
+ * virtual machine is presented a virtual GPU (vGPU), which has

equivalent
+ * features as the underlying physical GPU (pGPU), so i915 driver can run
+ * seamlessly in a virtual machine. This file provides vGPU specific
+ * optimizations when running in a virtual machine, to reduce the complexity
+ * of vGPU emulation and to improve the overall performance.
+ *
+ * A primary function introduced here is so-called "address space ballooning"
+ * technique. Intel GVT-g partitions global graphics memory among

Quoted 8 lines
Quoted 5 lines

multiple VMs,
+ * so each VM can directly access a portion of the memory w/o hypervisor's
+ * intervention, e.g. filling textures or queuing commands. However w/ the

Personal preference, but It may be nicer to avoid "w/o" and similar shorthand in documentation.

OK. Will fix it. :)

Quoted 43 lines
Quoted 43 lines

+ * partitioning an unmodified i915 driver would assume a smaller graphics
+ * memory starting from address ZERO, then requires vGPU emulation module to
+ * translate the graphics address between 'guest view' and 'host view', for
+ * all registers and command opcodes which contain a graphics memory address.
+ * To reduce the complexity, Intel GVT-g introduces "address space ballooning",
+ * by telling the exact partitioning knowledge to each guest i915 driver, which
+ * then reserves and prevents non-allocated portions from allocation. Thus vGPU
+ * emulation module module only needs to scan and validate graphics addresses
+ * w/o complexity of address translation.
+ *
+ */

drivers/gpu/drm/i915/i915_vgpu.c new file +6 -0
+
+/**
+ * i915_check_vgpu - detect virtual GPU
+ * @dev: drm device *
+ *
+ * This function is called at the initialization stage, to detect

whether
+ * running on a vGPU.
+ */

drivers/gpu/drm/i915/i915_vgpu.c new file +14 -0
+void i915_check_vgpu(struct drm_device *dev)
+{
+    struct drm_i915_private *dev_priv = to_i915(dev);
+    uint64_t magic;
+    uint32_t version;
+
+    BUILD_BUG_ON(sizeof(struct vgt_if) != VGT_PVINFO_SIZE);
+
+    if (!IS_HASWELL(dev))
+        return;
+
+    magic = readq(dev_priv->regs + vgtif_reg(magic));
+    if (magic != VGT_MAGIC)
+        return;
Quoted 8 lines

drivers/gpu/drm/i915/i915_vgpu.c line 81

Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +6 -0
+
+    version = INTEL_VGT_IF_VERSION_ENCODE(
+        readw(dev_priv->regs + vgtif_reg(version_major)),
+        readw(dev_priv->regs + vgtif_reg(version_minor)));
+    if (version != INTEL_VGT_IF_VERSION)
+        return;

Would it be useful to something on version mismatch?

Well. Thanks for your notification. :)
In fact, this version judgement is just for future usages. By now, we have only one vgt interface version, 0x10000. But you are right, unlike the judgement for 'magic', which specifies the Intel GVT-g feature, and can be returned directly when false, failure for the 'version' may should trigger something like unload the driver. However, if so, this means we may need to:
1> return something in i915_check_vgpu();
2> change the prototype of i915_check_vgpu() and its caller intel_uncore_init();
3> check the return value of intel_uncore_init() in i915_driver_load()
Is this approach acceptable?

Quoted 68 lines
Quoted 68 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +3 -0
+    dev_priv->vgpu.active = true;
+    DRM_INFO("Virtual GPU for Intel GVT-g detected.\n");
+}
drivers/gpu/drm/i915/i915_vgpu.h new file +4 -0
new file mode 100644
index 0000000..5f41d01c
@@ -0,0 +1,85 @@
+/*
+ * Copyright(c) 2011-2014 Intel Corporation. All rights reserved.
+ *
+ * Permission is hereby granted, free of charge, to any person

obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */

drivers/gpu/drm/i915/i915_vgpu.h new file +17 -0
+
+#ifndef _I915_VGPU_H_
+#define _I915_VGPU_H_
+
+/* The MMIO offset of the shared info between guest and host emulator */
+#define VGT_PVINFO_PAGE    0x78000
+#define VGT_PVINFO_SIZE    0x1000
+
+/*
+ * The following structure pages are defined in GEN MMIO space
+ * for virtualization. (One page for now)
+ */
+#define VGT_MAGIC         0x4776544776544776    /* 'vGTvGTvG' */
+#define VGT_VERSION_MAJOR 1
+#define VGT_VERSION_MINOR 0
+
+#define INTEL_VGT_IF_VERSION_ENCODE(major, minor) ((major) << 16 |

(minor))

drivers/gpu/drm/i915/i915_vgpu.h new file +5 -0
+#define INTEL_VGT_IF_VERSION \
+    INTEL_VGT_IF_VERSION_ENCODE(VGT_VERSION_MAJOR, VGT_VERSION_MINOR)
+
+struct vgt_if {
+    uint64_t magic;        /* VGT_MAGIC */
Quoted 9 lines

drivers/gpu/drm/i915/i915_vgpu.h line 48

Quoted 4 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +4 -0
+    uint16_t version_major;
+    uint16_t version_minor;
+    uint32_t vgt_id;    /* ID of vGT instance */
+    uint32_t rsv1[12];    /* pad to offset 0x40 */

In general I thought stdint types should be avoided in the kernel in favour of u32 and company. But I see i915 is full of them already so maybe my information is outdated.

Let's keep the _t types, like explanations in Daniel's mail. Is this OK?

Quoted 22 lines
Quoted 11 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +11 -0
+    /*
+     *  Data structure to describe the balooning info of resources.
+     *  Each VM can only have one portion of continuous area for now.
+     *  (May support scattered resource in future)
+     *  (starting from offset 0x40)
+     */
+    struct {
+        /* Aperture register balooning */
+        struct {
+            uint32_t my_base;
+            uint32_t my_size;

drivers/gpu/drm/i915/i915_vgpu.h line 69

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +10 -0
+        } low_gmadr;    /* aperture */
+        /* GMADR register balooning */
+        struct {
+            uint32_t my_base;
+            uint32_t my_size;
+        } high_gmadr;    /* non aperture */
+        /* allowed fence registers */
+        uint32_t fence_num;
+        uint32_t rsv2[3];
+    } avail_rs;        /* available/assigned resource */
Quoted 4 lines

I didn't figure out yet why the concept is to pass in two ranges to mark as reserved, instead of one usable/allowed range and then implicitly all the rest as reserved?

Like you said in 2/3 comments, this 2 part is for mappable and non-mapplable respectively. :)

Quoted 11 lines

drivers/gpu/drm/i915/i915_vgpu.h line 78

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +3 -0
+    uint32_t rsv3[0x200 - 24];    /* pad to half page */
+    /*
+     * The bottom half page is for response from Gfx driver to

hypervisor.

drivers/gpu/drm/i915/i915_vgpu.h new file +6 -0
+     * Set to reserved fields temporarily by now.
+     */
+    uint32_t rsv4;
+    uint32_t display_ready;    /* ready for display owner switch */
+    uint32_t rsv5[0x200 - 2];    /* pad to one page */
+} __packed;
Quoted 4 lines

You probably don't need __packed since you have BUILD_BUG_ON on size mismatch and pad to full size with reserved fields. I don't think it matters though.

Well, IIRC, this __packed is added to follow Chris's comments. How about we keep it? Or will it hurt any place? :)

Quoted 32 lines
Quoted 21 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +6 -0
+#define vgtif_reg(x) \
+    (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)
+
+extern void i915_check_vgpu(struct drm_device *dev);
+
+#endif /* _I915_VGPU_H_ */
drivers/gpu/drm/i915/intel_uncore.c +1 -0
index 9427641..cae27bb 100644
@@ -23,6 +23,7 @@
 
  #include "i915_drv.h"
  #include "intel_drv.h"
+#include "i915_vgpu.h"
 
  #define FORCEWAKE_ACK_TIMEOUT_MS 2
 

drivers/gpu/drm/i915/intel_uncore.c void intel_uncore_init(struct drm_device *dev) · line 858

Quoted 8 lines
drivers/gpu/drm/i915/intel_uncore.c +2 -0
@@ -850,6 +851,8 @@ void intel_uncore_init(struct drm_device *dev)
  {
      struct drm_i915_private *dev_priv = dev->dev_private;
 
+    i915_check_vgpu(dev);
+
      setup_timer(&dev_priv->uncore.force_wake_timer,
              gen6_force_wake_timer, (unsigned long)dev_priv);
 

Regards,

Quoted 1 line

Tvrtko

B.R.
Yu

Quoted 2 lines

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#5 lore raw

Re: [PATCH v3 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.

Quoted 11 lines · Zhang

Thank you very much for your thorough review, Tvrtko. :)

Quoted 8 lines · Tvrtko Ursulin
Quoted 5 lines · Yu Zhang

Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through. This page contains the shared information between i915 driver and the host emulator. For now, this structure utilizes an area of 4K bypes on HSW GPU's unused MMIO space to support

Quoted 8 lines
Quoted 1 line

bytes

Got it. Thanks.

Quoted 4 lines
Quoted 2 lines

existing production. Future hardware will have the reserved window

It is slightly unclear to me what "existing production" means?

Our current gpu virtualization implementation for HSW. It's compared with future implematations for BDW etc.

So maybe instead of "existing production" say "currently supported hardware" or similar?

Quoted 242 lines
Quoted 182 lines
Quoted 182 lines

architecturally defined, and layout of the page will be added in future BSpec.

The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. Thereafter a flag, vgpu.active is set, and intel_vgpu_active() is used by checking this flag to conclude if GPU is virtualized with Intel GVT-g. By now, intel_vgpu_active() will return true, only when the driver is running as a guest in the Intel GVT-g enhanced environment on HSW platform.

v2:
take Chris' comments:
- call the i915_check_vgpu() in intel_uncore_init()
- sanitize i915_check_vgpu() by adding BUILD_BUG_ON() and debug info
take Daniel's comments:
- put the definition of PV INFO into a new header - i915_vgt_if.h
other changes:
- access mmio regs by readq/readw in i915_check_vgpu()

v3:
take Daniel's comments:
- move the i915/vgt interfaces into a new i915_vgpu.c
- update makefile
- add kerneldoc to functions which are non-static
- add a DOC: section describing some of the high-level design
- update drm docbook
other changes:
- rename i915_vgt_if.h to i915_vgpu.h

---
  Documentation/DocBook/drm.tmpl      |  5 +++
  drivers/gpu/drm/i915/Makefile       |  3 ++
  drivers/gpu/drm/i915/i915_drv.h     | 11 +++++
  drivers/gpu/drm/i915/i915_vgpu.c    | 85
+++++++++++++++++++++++++++++++++++++
  drivers/gpu/drm/i915/i915_vgpu.h    | 85
+++++++++++++++++++++++++++++++++++++
  drivers/gpu/drm/i915/intel_uncore.c |  3 ++
  6 files changed, 192 insertions(+)

create mode 100644 drivers/gpu/drm/i915/i915_vgpu.c create mode 100644 drivers/gpu/drm/i915/i915_vgpu.h

Documentation/DocBook/drm.tmpl +5 -0
index 64d9c1e..e4db4cf 100644
@@ -3832,6 +3832,11 @@ int num_ioctls;</synopsis>
  !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_disable_interrupts
  !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_enable_interrupts
        </sect2>
+      <sect2>
+        <title>Intel GVT-g Guest Support(vGPU)</title>
+!Pdrivers/gpu/drm/i915/i915_vgpu.c Intel GVT-g guest support
+!Idrivers/gpu/drm/i915/i915_vgpu.c
+      </sect2>
      </sect1>
      <sect1>
        <title>Display Hardware Handling</title>
drivers/gpu/drm/i915/Makefile +3 -0
index 891e584..d9aa5ca 100644
@@ -79,6 +79,9 @@ i915-y += dvo_ch7017.o \
        intel_sdvo.o \
        intel_tv.o
 
+# virtual gpu code
+i915-y += i915_vgpu.o
+
  # legacy horrors
  i915-y += i915_dma.o \
        i915_ums.o
drivers/gpu/drm/i915/i915_drv.h +11 -0
index 54691bc..e1e221e 100644
@@ -1470,6 +1470,10 @@ struct i915_workarounds {
      u32 count;
  };
 
+struct i915_virtual_gpu {
+    bool active;
+};
+
  struct drm_i915_private {
      struct drm_device *dev;
      struct kmem_cache *slab;
@@ -1482,6 +1486,8 @@ struct drm_i915_private {
 
      struct intel_uncore uncore;
 
+    struct i915_virtual_gpu vgpu;
+
      struct intel_gmbus gmbus[GMBUS_NUM_PORTS];
 
 
@@ -2311,6 +2317,11 @@ extern void intel_uncore_check_errors(struct
 drm_device *dev);
  extern void intel_uncore_fini(struct drm_device *dev);
  extern void intel_uncore_forcewake_reset(struct drm_device *dev,
 bool restore);
 
+static inline bool intel_vgpu_active(struct drm_device *dev)
+{
+    return to_i915(dev)->vgpu.active;
+}
+
  void

i915_enable_pipestat(struct drm_i915_private *dev_priv, enum pipe
pipe,

               u32 status_mask);
drivers/gpu/drm/i915/i915_vgpu.c new file +4 -0
new file mode 100644
index 0000000..3f6b797
@@ -0,0 +1,85 @@
+/*
+ * Copyright(c) 2011-2014 Intel Corporation. All rights reserved.
+ *
+ * Permission is hereby granted, free of charge, to any person

obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */

drivers/gpu/drm/i915/i915_vgpu.c new file +11 -0
+
+#include "intel_drv.h"
+#include "i915_vgpu.h"
+
+
+/**
+ * DOC: Intel GVT-g guest support
+ *
+ * Intel GVT-g is a graphics virtualization technology which shares the
+ * GPU among multiple virtual machines on a time-sharing basis. Each
+ * virtual machine is presented a virtual GPU (vGPU), which has

equivalent
+ * features as the underlying physical GPU (pGPU), so i915 driver can run
+ * seamlessly in a virtual machine. This file provides vGPU specific
+ * optimizations when running in a virtual machine, to reduce the complexity
+ * of vGPU emulation and to improve the overall performance.
+ *
+ * A primary function introduced here is so-called "address space ballooning"
+ * technique. Intel GVT-g partitions global graphics memory among

Quoted 8 lines
Quoted 5 lines

multiple VMs,
+ * so each VM can directly access a portion of the memory w/o hypervisor's
+ * intervention, e.g. filling textures or queuing commands. However w/ the

Personal preference, but It may be nicer to avoid "w/o" and similar shorthand in documentation.

OK. Will fix it. :)

Quoted 43 lines
Quoted 43 lines

+ * partitioning an unmodified i915 driver would assume a smaller graphics
+ * memory starting from address ZERO, then requires vGPU emulation module to
+ * translate the graphics address between 'guest view' and 'host view', for
+ * all registers and command opcodes which contain a graphics memory address.
+ * To reduce the complexity, Intel GVT-g introduces "address space ballooning",
+ * by telling the exact partitioning knowledge to each guest i915 driver, which
+ * then reserves and prevents non-allocated portions from allocation. Thus vGPU
+ * emulation module module only needs to scan and validate graphics addresses
+ * w/o complexity of address translation.
+ *
+ */

drivers/gpu/drm/i915/i915_vgpu.c new file +6 -0
+
+/**
+ * i915_check_vgpu - detect virtual GPU
+ * @dev: drm device *
+ *
+ * This function is called at the initialization stage, to detect

whether
+ * running on a vGPU.
+ */

drivers/gpu/drm/i915/i915_vgpu.c new file +14 -0
+void i915_check_vgpu(struct drm_device *dev)
+{
+    struct drm_i915_private *dev_priv = to_i915(dev);
+    uint64_t magic;
+    uint32_t version;
+
+    BUILD_BUG_ON(sizeof(struct vgt_if) != VGT_PVINFO_SIZE);
+
+    if (!IS_HASWELL(dev))
+        return;
+
+    magic = readq(dev_priv->regs + vgtif_reg(magic));
+    if (magic != VGT_MAGIC)
+        return;
Quoted 8 lines

drivers/gpu/drm/i915/i915_vgpu.c line 81

Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +6 -0
+
+    version = INTEL_VGT_IF_VERSION_ENCODE(
+        readw(dev_priv->regs + vgtif_reg(version_major)),
+        readw(dev_priv->regs + vgtif_reg(version_minor)));
+    if (version != INTEL_VGT_IF_VERSION)
+        return;

Would it be useful to something on version mismatch?

Quoted 12 lines

Well. Thanks for your notification. :)
In fact, this version judgement is just for future usages. By now, we have only one vgt interface version, 0x10000. But you are right, unlike the judgement for 'magic', which specifies the Intel GVT-g feature, and can be returned directly when false, failure for the 'version' may should trigger something like unload the driver. However, if so, this means we may need to:
1> return something in i915_check_vgpu();
2> change the prototype of i915_check_vgpu() and its caller intel_uncore_init();
3> check the return value of intel_uncore_init() in i915_driver_load() Is this approach acceptable?

I would leave this for the future version. I was simply suggesting logging something on version mismatch. Just because it is unexpected so there is a trace in the logs why feature is not available. It's up to you.

Quoted 79 lines
Quoted 70 lines
Quoted 70 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +3 -0
+    dev_priv->vgpu.active = true;
+    DRM_INFO("Virtual GPU for Intel GVT-g detected.\n");
+}
drivers/gpu/drm/i915/i915_vgpu.h new file +4 -0
new file mode 100644
index 0000000..5f41d01c
@@ -0,0 +1,85 @@
+/*
+ * Copyright(c) 2011-2014 Intel Corporation. All rights reserved.
+ *
+ * Permission is hereby granted, free of charge, to any person

obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */

drivers/gpu/drm/i915/i915_vgpu.h new file +5 -0
+
+#ifndef _I915_VGPU_H_
+#define _I915_VGPU_H_
+
+/* The MMIO offset of the shared info between guest and host

emulator */

drivers/gpu/drm/i915/i915_vgpu.h new file +12 -0
+#define VGT_PVINFO_PAGE    0x78000
+#define VGT_PVINFO_SIZE    0x1000
+
+/*
+ * The following structure pages are defined in GEN MMIO space
+ * for virtualization. (One page for now)
+ */
+#define VGT_MAGIC         0x4776544776544776    /* 'vGTvGTvG' */
+#define VGT_VERSION_MAJOR 1
+#define VGT_VERSION_MINOR 0
+
+#define INTEL_VGT_IF_VERSION_ENCODE(major, minor) ((major) << 16 |

(minor))

drivers/gpu/drm/i915/i915_vgpu.h new file +5 -0
+#define INTEL_VGT_IF_VERSION \
+    INTEL_VGT_IF_VERSION_ENCODE(VGT_VERSION_MAJOR, VGT_VERSION_MINOR)
+
+struct vgt_if {
+    uint64_t magic;        /* VGT_MAGIC */
Quoted 9 lines

drivers/gpu/drm/i915/i915_vgpu.h line 48

Quoted 4 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +4 -0
+    uint16_t version_major;
+    uint16_t version_minor;
+    uint32_t vgt_id;    /* ID of vGT instance */
+    uint32_t rsv1[12];    /* pad to offset 0x40 */

In general I thought stdint types should be avoided in the kernel in favour of u32 and company. But I see i915 is full of them already so maybe my information is outdated.

Quoted 1 line

Let's keep the _t types, like explanations in Daniel's mail. Is this OK?

Yes.

Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +21 -0
+    /*
+     *  Data structure to describe the balooning info of resources.
+     *  Each VM can only have one portion of continuous area for now.
+     *  (May support scattered resource in future)
+     *  (starting from offset 0x40)
+     */
+    struct {
+        /* Aperture register balooning */
+        struct {
+            uint32_t my_base;
+            uint32_t my_size;
+        } low_gmadr;    /* aperture */
+        /* GMADR register balooning */
+        struct {
+            uint32_t my_base;
+            uint32_t my_size;
+        } high_gmadr;    /* non aperture */
+        /* allowed fence registers */
+        uint32_t fence_num;
+        uint32_t rsv2[3];
+    } avail_rs;        /* available/assigned resource */
Quoted 6 lines
Quoted 4 lines

I didn't figure out yet why the concept is to pass in two ranges to mark as reserved, instead of one usable/allowed range and then implicitly all the rest as reserved?

Like you said in 2/3 comments, this 2 part is for mappable and non-mapplable respectively. :)

Are high and low memory as terms known in the i915 space so far as synonyms for mappable and non-mappable?

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +3 -0
+    uint32_t rsv3[0x200 - 24];    /* pad to half page */
+    /*
+     * The bottom half page is for response from Gfx driver to

hypervisor.

drivers/gpu/drm/i915/i915_vgpu.h new file +6 -0
+     * Set to reserved fields temporarily by now.
+     */
+    uint32_t rsv4;
+    uint32_t display_ready;    /* ready for display owner switch */
+    uint32_t rsv5[0x200 - 2];    /* pad to one page */
+} __packed;
Quoted 6 lines
Quoted 4 lines

You probably don't need __packed since you have BUILD_BUG_ON on size mismatch and pad to full size with reserved fields. I don't think it matters though.

Well, IIRC, this __packed is added to follow Chris's comments. How about we keep it? Or will it hurt any place? :)

The only purpose for packed here I can see is a double guard along the BUILD_BUG_ON. As I said I don't think it makes any difference so you can leave it in.

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#6 lore raw

Re: [PATCH v3 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.

Quoted 16 lines · Tvrtko Ursulin
Quoted 14 lines · Zhang

Thank you very much for your thorough review, Tvrtko. :)

Quoted 9 lines · Tvrtko Ursulin
Quoted 5 lines · Yu Zhang

Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through. This page contains the shared information between i915 driver and the host emulator. For now, this structure utilizes an area of 4K bypes on HSW GPU's unused MMIO space to support

bytes

Got it. Thanks.

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

existing production. Future hardware will have the reserved window

It is slightly unclear to me what "existing production" means?

Our current gpu virtualization implementation for HSW. It's compared with future implematations for BDW etc.

So maybe instead of "existing production" say "currently supported hardware" or similar?

How about we remove "to support existing production" in this sentence? The "currently supported hardware" seems a bit redundant. :)

Quoted 373 lines
Quoted 246 lines
Quoted 186 lines
Quoted 186 lines

architecturally defined, and layout of the page will be added in future BSpec.

The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. Thereafter a flag, vgpu.active is set, and intel_vgpu_active() is used by checking this flag to conclude if GPU is virtualized with Intel GVT-g. By now, intel_vgpu_active() will return true, only when the driver is running as a guest in the Intel GVT-g enhanced environment on HSW platform.

v2:
take Chris' comments:
- call the i915_check_vgpu() in intel_uncore_init()
- sanitize i915_check_vgpu() by adding BUILD_BUG_ON() and debug
info
take Daniel's comments:
- put the definition of PV INFO into a new header - i915_vgt_if.h
other changes:
- access mmio regs by readq/readw in i915_check_vgpu()

v3:
take Daniel's comments:
- move the i915/vgt interfaces into a new i915_vgpu.c
- update makefile
- add kerneldoc to functions which are non-static
- add a DOC: section describing some of the high-level design
- update drm docbook
other changes:
- rename i915_vgt_if.h to i915_vgpu.h

---
  Documentation/DocBook/drm.tmpl      |  5 +++
  drivers/gpu/drm/i915/Makefile       |  3 ++
  drivers/gpu/drm/i915/i915_drv.h     | 11 +++++
  drivers/gpu/drm/i915/i915_vgpu.c    | 85
+++++++++++++++++++++++++++++++++++++
  drivers/gpu/drm/i915/i915_vgpu.h    | 85
+++++++++++++++++++++++++++++++++++++
  drivers/gpu/drm/i915/intel_uncore.c |  3 ++
  6 files changed, 192 insertions(+)

create mode 100644 drivers/gpu/drm/i915/i915_vgpu.c create mode 100644 drivers/gpu/drm/i915/i915_vgpu.h

Documentation/DocBook/drm.tmpl +5 -0
index 64d9c1e..e4db4cf 100644
@@ -3832,6 +3832,11 @@ int num_ioctls;</synopsis>
  !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_disable_interrupts
  !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_enable_interrupts
        </sect2>
+      <sect2>
+        <title>Intel GVT-g Guest Support(vGPU)</title>
+!Pdrivers/gpu/drm/i915/i915_vgpu.c Intel GVT-g guest support
+!Idrivers/gpu/drm/i915/i915_vgpu.c
+      </sect2>
      </sect1>
      <sect1>
        <title>Display Hardware Handling</title>
drivers/gpu/drm/i915/Makefile +3 -0
index 891e584..d9aa5ca 100644
@@ -79,6 +79,9 @@ i915-y += dvo_ch7017.o \
        intel_sdvo.o \
        intel_tv.o
 
+# virtual gpu code
+i915-y += i915_vgpu.o
+
  # legacy horrors
  i915-y += i915_dma.o \
        i915_ums.o
drivers/gpu/drm/i915/i915_drv.h +11 -0
index 54691bc..e1e221e 100644
@@ -1470,6 +1470,10 @@ struct i915_workarounds {
      u32 count;
  };
 
+struct i915_virtual_gpu {
+    bool active;
+};
+
  struct drm_i915_private {
      struct drm_device *dev;
      struct kmem_cache *slab;
@@ -1482,6 +1486,8 @@ struct drm_i915_private {
 
      struct intel_uncore uncore;
 
+    struct i915_virtual_gpu vgpu;
+
      struct intel_gmbus gmbus[GMBUS_NUM_PORTS];
 
 
@@ -2311,6 +2317,11 @@ extern void intel_uncore_check_errors(struct
 drm_device *dev);
  extern void intel_uncore_fini(struct drm_device *dev);
  extern void intel_uncore_forcewake_reset(struct drm_device *dev,
 bool restore);
 
+static inline bool intel_vgpu_active(struct drm_device *dev)
+{
+    return to_i915(dev)->vgpu.active;
+}
+
  void

i915_enable_pipestat(struct drm_i915_private *dev_priv, enum pipe
pipe,

               u32 status_mask);
drivers/gpu/drm/i915/i915_vgpu.c new file +4 -0
new file mode 100644
index 0000000..3f6b797
@@ -0,0 +1,85 @@
+/*
+ * Copyright(c) 2011-2014 Intel Corporation. All rights reserved.
+ *
+ * Permission is hereby granted, free of charge, to any person

obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */

drivers/gpu/drm/i915/i915_vgpu.c new file +9 -0
+
+#include "intel_drv.h"
+#include "i915_vgpu.h"
+
+
+/**
+ * DOC: Intel GVT-g guest support
+ *
+ * Intel GVT-g is a graphics virtualization technology which shares

the
+ * GPU among multiple virtual machines on a time-sharing basis. Each
+ * virtual machine is presented a virtual GPU (vGPU), which has equivalent
+ * features as the underlying physical GPU (pGPU), so i915 driver can run
+ * seamlessly in a virtual machine. This file provides vGPU specific
+ * optimizations when running in a virtual machine, to reduce the complexity
+ * of vGPU emulation and to improve the overall performance.
+ *
+ * A primary function introduced here is so-called "address space ballooning"
+ * technique. Intel GVT-g partitions global graphics memory among

Quoted 8 lines
Quoted 5 lines

multiple VMs,
+ * so each VM can directly access a portion of the memory w/o hypervisor's
+ * intervention, e.g. filling textures or queuing commands. However w/ the

Personal preference, but It may be nicer to avoid "w/o" and similar shorthand in documentation.

OK. Will fix it. :)

Quoted 43 lines
Quoted 43 lines

+ * partitioning an unmodified i915 driver would assume a smaller graphics
+ * memory starting from address ZERO, then requires vGPU emulation module to
+ * translate the graphics address between 'guest view' and 'host view', for
+ * all registers and command opcodes which contain a graphics memory address.
+ * To reduce the complexity, Intel GVT-g introduces "address space ballooning",
+ * by telling the exact partitioning knowledge to each guest i915 driver, which
+ * then reserves and prevents non-allocated portions from allocation. Thus vGPU
+ * emulation module module only needs to scan and validate graphics addresses
+ * w/o complexity of address translation.
+ *
+ */

drivers/gpu/drm/i915/i915_vgpu.c new file +6 -0
+
+/**
+ * i915_check_vgpu - detect virtual GPU
+ * @dev: drm device *
+ *
+ * This function is called at the initialization stage, to detect

whether
+ * running on a vGPU.
+ */

drivers/gpu/drm/i915/i915_vgpu.c new file +14 -0
+void i915_check_vgpu(struct drm_device *dev)
+{
+    struct drm_i915_private *dev_priv = to_i915(dev);
+    uint64_t magic;
+    uint32_t version;
+
+    BUILD_BUG_ON(sizeof(struct vgt_if) != VGT_PVINFO_SIZE);
+
+    if (!IS_HASWELL(dev))
+        return;
+
+    magic = readq(dev_priv->regs + vgtif_reg(magic));
+    if (magic != VGT_MAGIC)
+        return;
Quoted 8 lines

drivers/gpu/drm/i915/i915_vgpu.c line 81

Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +6 -0
+
+    version = INTEL_VGT_IF_VERSION_ENCODE(
+        readw(dev_priv->regs + vgtif_reg(version_major)),
+        readw(dev_priv->regs + vgtif_reg(version_minor)));
+    if (version != INTEL_VGT_IF_VERSION)
+        return;

Would it be useful to something on version mismatch?

Quoted 12 lines

Well. Thanks for your notification. :)
In fact, this version judgement is just for future usages. By now, we have only one vgt interface version, 0x10000. But you are right, unlike the judgement for 'magic', which specifies the Intel GVT-g feature, and can be returned directly when false, failure for the 'version' may should trigger something like unload the driver. However, if so, this means we may need to:
1> return something in i915_check_vgpu();
2> change the prototype of i915_check_vgpu() and its caller intel_uncore_init();
3> check the return value of intel_uncore_init() in i915_driver_load() Is this approach acceptable?

I would leave this for the future version. I was simply suggesting logging something on version mismatch. Just because it is unexpected so there is a trace in the logs why feature is not available. It's up to you.

Quoted 80 lines
Quoted 71 lines
Quoted 71 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +3 -0
+    dev_priv->vgpu.active = true;
+    DRM_INFO("Virtual GPU for Intel GVT-g detected.\n");
+}
drivers/gpu/drm/i915/i915_vgpu.h new file +4 -0
new file mode 100644
index 0000000..5f41d01c
@@ -0,0 +1,85 @@
+/*
+ * Copyright(c) 2011-2014 Intel Corporation. All rights reserved.
+ *
+ * Permission is hereby granted, free of charge, to any person

obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */

drivers/gpu/drm/i915/i915_vgpu.h new file +5 -0
+
+#ifndef _I915_VGPU_H_
+#define _I915_VGPU_H_
+
+/* The MMIO offset of the shared info between guest and host

emulator */

drivers/gpu/drm/i915/i915_vgpu.h new file +12 -0
+#define VGT_PVINFO_PAGE    0x78000
+#define VGT_PVINFO_SIZE    0x1000
+
+/*
+ * The following structure pages are defined in GEN MMIO space
+ * for virtualization. (One page for now)
+ */
+#define VGT_MAGIC         0x4776544776544776    /* 'vGTvGTvG' */
+#define VGT_VERSION_MAJOR 1
+#define VGT_VERSION_MINOR 0
+
+#define INTEL_VGT_IF_VERSION_ENCODE(major, minor) ((major) << 16 |

(minor))

drivers/gpu/drm/i915/i915_vgpu.h new file +5 -0
+#define INTEL_VGT_IF_VERSION \
+    INTEL_VGT_IF_VERSION_ENCODE(VGT_VERSION_MAJOR, VGT_VERSION_MINOR)
+
+struct vgt_if {
+    uint64_t magic;        /* VGT_MAGIC */
Quoted 9 lines

drivers/gpu/drm/i915/i915_vgpu.h line 48

Quoted 4 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +4 -0
+    uint16_t version_major;
+    uint16_t version_minor;
+    uint32_t vgt_id;    /* ID of vGT instance */
+    uint32_t rsv1[12];    /* pad to offset 0x40 */

In general I thought stdint types should be avoided in the kernel in favour of u32 and company. But I see i915 is full of them already so maybe my information is outdated.

Quoted 1 line

Let's keep the _t types, like explanations in Daniel's mail. Is this OK?

Yes.

Quoted 26 lines
Quoted 26 lines
Quoted 11 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +11 -0
+    /*
+     *  Data structure to describe the balooning info of resources.
+     *  Each VM can only have one portion of continuous area for now.
+     *  (May support scattered resource in future)
+     *  (starting from offset 0x40)
+     */
+    struct {
+        /* Aperture register balooning */
+        struct {
+            uint32_t my_base;
+            uint32_t my_size;

drivers/gpu/drm/i915/i915_vgpu.h line 69

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +10 -0
+        } low_gmadr;    /* aperture */
+        /* GMADR register balooning */
+        struct {
+            uint32_t my_base;
+            uint32_t my_size;
+        } high_gmadr;    /* non aperture */
+        /* allowed fence registers */
+        uint32_t fence_num;
+        uint32_t rsv2[3];
+    } avail_rs;        /* available/assigned resource */

I didn't figure out yet why the concept is to pass in two ranges to mark as reserved, instead of one usable/allowed range and then implicitly all the rest as reserved?

Quoted 6 lines
Quoted 2 lines

Like you said in 2/3 comments, this 2 part is for mappable and non-mapplable respectively. :)

Are high and low memory as terms known in the i915 space so far as synonyms for mappable and non-mappable?

Well, I'm not sure.
Daniel, do you think the "low gmadr/high gmadr" will cause any misunderstanding?

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

drivers/gpu/drm/i915/i915_vgpu.h line 78

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +3 -0
+    uint32_t rsv3[0x200 - 24];    /* pad to half page */
+    /*
+     * The bottom half page is for response from Gfx driver to

hypervisor.

drivers/gpu/drm/i915/i915_vgpu.h new file +6 -0
+     * Set to reserved fields temporarily by now.
+     */
+    uint32_t rsv4;
+    uint32_t display_ready;    /* ready for display owner switch */
+    uint32_t rsv5[0x200 - 2];    /* pad to one page */
+} __packed;

You probably don't need __packed since you have BUILD_BUG_ON on size mismatch and pad to full size with reserved fields. I don't think it matters though.

Quoted 6 lines
Quoted 2 lines

Well, IIRC, this __packed is added to follow Chris's comments. How about we keep it? Or will it hurt any place? :)

The only purpose for packed here I can see is a double guard along the BUILD_BUG_ON. As I said I don't think it makes any difference so you can leave it in.

OK. Thanks.

Quoted 6 lines

Regards,

Tvrtko

B.R.
Yu
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#7 lore raw

Re: [PATCH v3 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.

Quoted 32 lines

From: Yu, Zhang
Sent: Wednesday, December 17, 2014 10:50 AM

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

drivers/gpu/drm/i915/i915_vgpu.h

Quoted 3 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +3 -0
+    /*
+     *  Data structure to describe the balooning info of resources.
+     *  Each VM can only have one portion of continuous area for

now.

Quoted 26 lines
Quoted 25 lines
Quoted 19 lines
Quoted 8 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +8 -0
+     *  (May support scattered resource in future)
+     *  (starting from offset 0x40)
+     */
+    struct {
+        /* Aperture register balooning */
+        struct {
+            uint32_t my_base;
+            uint32_t my_size;

drivers/gpu/drm/i915/i915_vgpu.h line 69

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +10 -0
+        } low_gmadr;    /* aperture */
+        /* GMADR register balooning */
+        struct {
+            uint32_t my_base;
+            uint32_t my_size;
+        } high_gmadr;    /* non aperture */
+        /* allowed fence registers */
+        uint32_t fence_num;
+        uint32_t rsv2[3];
+    } avail_rs;        /* available/assigned resource */
Quoted 4 lines

I didn't figure out yet why the concept is to pass in two ranges to mark as reserved, instead of one usable/allowed range and then implicitly all the rest as reserved?

Like you said in 2/3 comments, this 2 part is for mappable and non-mapplable respectively. :)

Quoted 6 lines
Quoted 3 lines

Are high and low memory as terms known in the i915 space so far as synonyms for mappable and non-mappable?

Well, I'm not sure.
Daniel, do you think the "low gmadr/high gmadr" will cause any misunderstanding?

for i915 possibly mappable and non-mappable are more intuitive, but for end user they finally need to know mappable is actually in low end and non-mappable is in high end, so low/high would be simpler to them. since this is the information used between host and guest, we're open to whatever meaningful to i915, and our host vgt driver can still keep low/high notations to user.

Thanks
Kevin
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#8 lore raw

PATCH v3 2/8 drm/i915: Adds graphic address space ballooning logic

In reply to Yu Zhang

With Intel GVT-g, the global graphic memory space is partitioned by multiple vGPU instances in different VMs. The ballooning code is called in i915_gem_setup_global_gtt(), utilizing the drm mm allocator APIs to mark the graphic address space which are partitioned out to other vGPUs as reserved.

v2:
take Chris and Daniel's comments:
- no guard page between different VMs
- use drm_mm_reserve_node() to do the reservation for ballooning, instead of the previous drm_mm_insert_node_in_range_generic()

v3:
take Daniel's comments:
- move ballooning functions into i915_vgpu.c
- add kerneldoc to ballooning functions

---
 drivers/gpu/drm/i915/i915_gem_gtt.c |  17 +++-
 drivers/gpu/drm/i915/i915_vgpu.c    | 149 ++++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_vgpu.h    |   2 +
 3 files changed, 165 insertions(+), 3 deletions(-)
drivers/gpu/drm/i915/i915_gem_gtt.c +14 -3
index de12017..2dfac13 100644
@@ -27,6 +27,7 @@
 #include <drm/drmP.h>
 #include <drm/i915_drm.h>
 #include "i915_drv.h"
+#include "i915_vgpu.h"
 #include "i915_trace.h"
 #include "intel_drv.h"
 
@@ -1683,6 +1684,16 @@ int i915_gem_setup_global_gtt(struct drm_device *dev,
 
 	/* Subtract the guard page ... */
 	drm_mm_init(&ggtt_vm->mm, start, end - start - PAGE_SIZE);
+
+	dev_priv->gtt.base.start = start;
+	dev_priv->gtt.base.total = end - start;
+
+	if (intel_vgpu_active(dev)) {
+		ret = intel_vgt_balloon(dev);
+		if (ret)
+			return ret;
+	}
+
 	if (!HAS_LLC(dev))
 		dev_priv->gtt.base.mm.color_adjust = i915_gtt_color_adjust;
 
@@ -1702,9 +1713,6 @@ int i915_gem_setup_global_gtt(struct drm_device *dev,
 		vma->bound |= GLOBAL_BIND;
 	}
 
-	dev_priv->gtt.base.start = start;
-	dev_priv->gtt.base.total = end - start;
-
 	/* Clear any non-preallocated blocks */
 	drm_mm_for_each_hole(entry, &ggtt_vm->mm, hole_start, hole_end) {
 		DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",
@@ -1756,6 +1764,9 @@ void i915_global_gtt_cleanup(struct drm_device *dev)
 	}
 
 	if (drm_mm_initialized(&vm->mm)) {
+		if (intel_vgpu_active(dev))
+			intel_vgt_deballoon();
+
 		drm_mm_takedown(&vm->mm);
 		list_del(&vm->global_link);
 	}
drivers/gpu/drm/i915/i915_vgpu.c +149 -0

152 lines not on this page · raw message

drivers/gpu/drm/i915/i915_vgpu.h +2 -0
index 5f41d01c..f538b18 100644
@@ -81,5 +81,7 @@ struct vgt_if {
 	(VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)
 
 extern void i915_check_vgpu(struct drm_device *dev);
+extern int intel_vgt_balloon(struct drm_device *dev);
+extern void intel_vgt_deballoon(void);
 
 #endif /* _I915_VGPU_H_ */
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#9 lore raw

Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 176

Quoted 4 lines · Yu Zhang
drivers/gpu/drm/i915/i915_vgpu.c +4 -0
+	if (low_gm_base < ggtt_vm->start
+	    || low_gm_end > dev_priv->gtt.mappable_end
+	    || high_gm_base < dev_priv->gtt.mappable_end
+	    || high_gm_end > ggtt_vm_end) {

Nit: Logical operators like || or && should be at the end of the previous line, not at the beginning of the new line. checkpatch would have noticed this one.
-Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#10 lore raw

Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic

Quoted 11 lines · Daniel Vetter

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 176

Quoted 4 lines · Yu Zhang
drivers/gpu/drm/i915/i915_vgpu.c +4 -0
+	if (low_gm_base < ggtt_vm->start
+	    || low_gm_end > dev_priv->gtt.mappable_end
+	    || high_gm_base < dev_priv->gtt.mappable_end
+	    || high_gm_end > ggtt_vm_end) {

Nit: Logical operators like || or && should be at the end of the previous line, not at the beginning of the new line. checkpatch would have noticed this one.
-Daniel

Got it.
Thank you. :-)

Yu
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#11 lore raw

Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic

In reply to Yu Zhang
Quoted 43 lines · Yu Zhang

With Intel GVT-g, the global graphic memory space is partitioned by multiple vGPU instances in different VMs. The ballooning code is called in i915_gem_setup_global_gtt(), utilizing the drm mm allocator APIs to mark the graphic address space which are partitioned out to other vGPUs as reserved.

v2:
take Chris and Daniel's comments:
- no guard page between different VMs
- use drm_mm_reserve_node() to do the reservation for ballooning, instead of the previous drm_mm_insert_node_in_range_generic()

v3:
take Daniel's comments:
- move ballooning functions into i915_vgpu.c
- add kerneldoc to ballooning functions

---
  drivers/gpu/drm/i915/i915_gem_gtt.c |  17 +++-
  drivers/gpu/drm/i915/i915_vgpu.c    | 149 ++++++++++++++++++++++++++++++++++++
  drivers/gpu/drm/i915/i915_vgpu.h    |   2 +
  3 files changed, 165 insertions(+), 3 deletions(-)
drivers/gpu/drm/i915/i915_gem_gtt.c +1 -0
index de12017..2dfac13 100644
@@ -27,6 +27,7 @@
  #include <drm/drmP.h>
  #include <drm/i915_drm.h>
  #include "i915_drv.h"
+#include "i915_vgpu.h"
  #include "i915_trace.h"
  #include "intel_drv.h"
 
@@ -1683,6 +1684,16 @@ int i915_gem_setup_global_gtt(struct drm_device *dev,
 
  	/* Subtract the guard page ... */
  	drm_mm_init(&ggtt_vm->mm, start, end - start - PAGE_SIZE);

drivers/gpu/drm/i915/i915_gem_gtt.c int i915_gem_setup_global_gtt(struct drm_device *dev, · line 1696

Quoted 10 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +10 -0
+
+	dev_priv->gtt.base.start = start;
+	dev_priv->gtt.base.total = end - start;
+
+	if (intel_vgpu_active(dev)) {
+		ret = intel_vgt_balloon(dev);
+		if (ret)
+			return ret;
+	}
+

Out of curiosity, what will be the mechanism to prevent a vGPU instance from ignoring the ballooning data? Must be something in the hypervisor blocking pass-through access to such domains?

And probably GPU reset should also be disallowed for vGPU instances?

Quoted 30 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +3 -3
  	if (!HAS_LLC(dev))
  		dev_priv->gtt.base.mm.color_adjust = i915_gtt_color_adjust;
 
@@ -1702,9 +1713,6 @@ int i915_gem_setup_global_gtt(struct drm_device *dev,
  		vma->bound |= GLOBAL_BIND;
  	}
 
-	dev_priv->gtt.base.start = start;
-	dev_priv->gtt.base.total = end - start;
-
  	/* Clear any non-preallocated blocks */
  	drm_mm_for_each_hole(entry, &ggtt_vm->mm, hole_start, hole_end) {
  		DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",
@@ -1756,6 +1764,9 @@ void i915_global_gtt_cleanup(struct drm_device *dev)
  	}
 
  	if (drm_mm_initialized(&vm->mm)) {
+		if (intel_vgpu_active(dev))
+			intel_vgt_deballoon();
+
  		drm_mm_takedown(&vm->mm);
  		list_del(&vm->global_link);
  	}
drivers/gpu/drm/i915/i915_vgpu.c +0 -0
index 3f6b797..ff5fba3 100644
@@ -83,3 +83,152 @@ void i915_check_vgpu(struct drm_device *dev)
  	dev_priv->vgpu.active = true;
  	DRM_INFO("Virtual GPU for Intel GVT-g detected.\n");

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 94

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +9 -0
  }
+
+struct _balloon_info_ {
+	/*
+	 * There are up to 2 regions per low/high graphic memory that
+	 * might be ballooned. Here, index 0/1 is for low
+	 * graphic memory, 2/3 for high graphic memory.
+	 */
+	struct drm_mm_node space[4];
+} bl_info;

This should be static I think.

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 106

Quoted 11 lines
drivers/gpu/drm/i915/i915_vgpu.c +11 -0
+/**
+ * intel_vgt_deballoon - deballoon reserved graphics address trunks
+ *
+ * This function is called to deallocate the ballooned-out graphic memory, when
+ * driver is unloaded or when ballooning fails.
+ */
+void intel_vgt_deballoon(void)
+{
+	int i;
+
+	DRM_INFO("VGT deballoon.\n");

Would debug be more appropriate? I don't see much value of saying this on driver unload - it's not that it is optional at this point.

Also for all logging, is intended human readable name VGT or vGT? If the latter it would be nicer to log it in that form.

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10 -0
+
+	for (i = 0; i < 4; i++) {
+		if (bl_info.space[i].allocated)
+			drm_mm_remove_node(&bl_info.space[i]);
+	}
+
+	memset(&bl_info, 0, sizeof(bl_info));
+}
+
+static int vgt_balloon_space(struct drm_mm *mm,

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 126

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10 -0
+			     struct drm_mm_node *node,
+			     unsigned long start, unsigned long end)
+{
+	unsigned long size = end - start;
+
+	if (start == end)
+		return -EINVAL;
+
+	DRM_INFO("balloon space: range [ 0x%lx - 0x%lx ] %lu KB.\n",
+		 start, end, size / 1024);

KiB ?

Quoted 5 lines
drivers/gpu/drm/i915/i915_vgpu.c +5 -0
+	node->start = start;
+	node->size = size;
+
+	return drm_mm_reserve_node(mm, node);
+}

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 142

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10 -0
+
+/**
+ * intel_vgt_balloon - balloon out reserved graphics address trunks
+ * @dev: drm device
+ *
+ * This function is called at the initialization stage, to balloon out the
+ * graphic address space allocated to other VMs, by marking these spaces as
+ * reserved.
+ *
+ * The ballooning related knowledges(starting address and size of the low/high

s/knowledges\(/knowledge /

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10 -0
+ * graphic memory) are depicted in the vgt_if structure in a reserved MMIO
+ * range.
+ *
+ * Returns:
+ * zero on success, non-zero if configuration invalid or ballooning failed
+ */
+int intel_vgt_balloon(struct drm_device *dev)
+{
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_address_space *ggtt_vm = &dev_priv->gtt.base;

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 162

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10 -0
+	unsigned long ggtt_vm_end = ggtt_vm->start + ggtt_vm->total;
+
+	unsigned long low_gm_base, low_gm_size, low_gm_end;
+	unsigned long high_gm_base, high_gm_size, high_gm_end;
+	int ret;
+
+	low_gm_base = I915_READ(vgtif_reg(avail_rs.low_gmadr.my_base));
+	low_gm_size = I915_READ(vgtif_reg(avail_rs.low_gmadr.my_size));
+	high_gm_base = I915_READ(vgtif_reg(avail_rs.high_gmadr.my_base));
+	high_gm_size = I915_READ(vgtif_reg(avail_rs.high_gmadr.my_size));

Get rid of my_ prefix ?

Quoted 9 lines
drivers/gpu/drm/i915/i915_vgpu.c +9 -0
+
+	low_gm_end = low_gm_base + low_gm_size;
+	high_gm_end = high_gm_base + high_gm_size;
+
+	DRM_INFO("VGT ballooning configuration:\n");
+	DRM_INFO("Low graphic memory: base 0x%lx size %ldKB\n",
+		 low_gm_base, low_gm_size / 1024);
+	DRM_INFO("High graphic memory: base 0x%lx size %ldKB\n",
+		 high_gm_base, high_gm_size / 1024);

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 181

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10 -0
+
+	if (low_gm_base < ggtt_vm->start
+	    || low_gm_end > dev_priv->gtt.mappable_end
+	    || high_gm_base < dev_priv->gtt.mappable_end
+	    || high_gm_end > ggtt_vm_end) {
+		DRM_ERROR("Invalid ballooning configuration!\n");
+		return -EINVAL;
+	}
+
+	memset(&bl_info, 0, sizeof(bl_info));

If bl_info is static then you don't need this memset?

Quoted 33 lines
drivers/gpu/drm/i915/i915_vgpu.c +33 -0
+	/* High graphic memory ballooning */
+	if (high_gm_base > dev_priv->gtt.mappable_end) {
+		ret = vgt_balloon_space(&ggtt_vm->mm,
+					&bl_info.space[2],
+					dev_priv->gtt.mappable_end,
+					high_gm_base);
+
+		if (ret)
+			goto err;
+	}
+
+	/*
+	 * No need to partition out the last physical page,
+	 * because it is reserved to the guard page.
+	 */
+	if (high_gm_end < ggtt_vm_end - PAGE_SIZE) {
+		ret = vgt_balloon_space(&ggtt_vm->mm,
+					&bl_info.space[3],
+					high_gm_end,
+					ggtt_vm_end - PAGE_SIZE);
+		if (ret)
+			goto err;
+	}
+
+	/* Low graphic memory ballooning */
+	if (low_gm_base > ggtt_vm->start) {
+		ret = vgt_balloon_space(&ggtt_vm->mm,
+					&bl_info.space[0],
+					ggtt_vm->start, low_gm_base);
+
+		if (ret)
+			goto err;
+	}

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 225

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10 -0
+
+	if (low_gm_end < dev_priv->gtt.mappable_end) {
+		ret = vgt_balloon_space(&ggtt_vm->mm,
+					&bl_info.space[1],
+					low_gm_end,
+					dev_priv->gtt.mappable_end);
+
+		if (ret)
+			goto err;
+	}

Okay, I've figured it out. :) I suppose going back to patch 1, where it says "Each VM can only have one portion of continuous area for now", with the emphasis on _one_. That threw me off thinking you have two "balloons" meaning forbidden areas. And then with low and high naming I got the wrong idea that one balloon marks the bottom inaccessible part, and the other top. I didn't figure out the whole low == mappable, high == non-mappable split. I suppose it was my inexperience, but if you can think of a way on how to improve the comment, even ASCII art would be very nice if possible.

Actually ballooning in the first place confuses me because I thought prior use for that in virtualisation was for memory which can shrink and grow at runtime. Did I get that wrong?

Quoted 14 lines
drivers/gpu/drm/i915/i915_vgpu.c +8 -0
+	DRM_INFO("VGT balloon successfully\n");
+	return 0;
+
+err:
+	DRM_ERROR("VGT balloon fail\n");
+	intel_vgt_deballoon();
+	return ret;
+}
drivers/gpu/drm/i915/i915_vgpu.h +0 -0
index 5f41d01c..f538b18 100644

drivers/gpu/drm/i915/i915_vgpu.h struct vgt_if { · line 87

Quoted 7 lines
drivers/gpu/drm/i915/i915_vgpu.h +2 -0
@@ -81,5 +81,7 @@ struct vgt_if {
  	(VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)
 
  extern void i915_check_vgpu(struct drm_device *dev);
+extern int intel_vgt_balloon(struct drm_device *dev);
+extern void intel_vgt_deballoon(void);
 
  #endif /* _I915_VGPU_H_ */

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#12 lore raw

Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic

Quoted 46 lines · Tvrtko Ursulin
Quoted 44 lines · Yu Zhang

With Intel GVT-g, the global graphic memory space is partitioned by multiple vGPU instances in different VMs. The ballooning code is called in i915_gem_setup_global_gtt(), utilizing the drm mm allocator APIs to mark the graphic address space which are partitioned out to other vGPUs as reserved.

v2:
take Chris and Daniel's comments:
- no guard page between different VMs
- use drm_mm_reserve_node() to do the reservation for ballooning, instead of the previous drm_mm_insert_node_in_range_generic()

v3:
take Daniel's comments:
- move ballooning functions into i915_vgpu.c
- add kerneldoc to ballooning functions

---
  drivers/gpu/drm/i915/i915_gem_gtt.c |  17 +++-
  drivers/gpu/drm/i915/i915_vgpu.c    | 149
++++++++++++++++++++++++++++++++++++
  drivers/gpu/drm/i915/i915_vgpu.h    |   2 +
  3 files changed, 165 insertions(+), 3 deletions(-)
drivers/gpu/drm/i915/i915_gem_gtt.c +1 -0
index de12017..2dfac13 100644
@@ -27,6 +27,7 @@
  #include <drm/drmP.h>
  #include <drm/i915_drm.h>
  #include "i915_drv.h"
+#include "i915_vgpu.h"
  #include "i915_trace.h"
  #include "intel_drv.h"
 
@@ -1683,6 +1684,16 @@ int i915_gem_setup_global_gtt(struct drm_device
 *dev,
 
Quoted 16 lines

drivers/gpu/drm/i915/i915_gem_gtt.c int i915_gem_setup_global_gtt(struct drm_device *dev, · line 1696

Quoted 12 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +10 -0
      /* Subtract the guard page ... */
      drm_mm_init(&ggtt_vm->mm, start, end - start - PAGE_SIZE);
+
+    dev_priv->gtt.base.start = start;
+    dev_priv->gtt.base.total = end - start;
+
+    if (intel_vgpu_active(dev)) {
+        ret = intel_vgt_balloon(dev);
+        if (ret)
+            return ret;
+    }
+

Out of curiosity, what will be the mechanism to prevent a vGPU instance from ignoring the ballooning data? Must be something in the hypervisor blocking pass-through access to such domains?

Well, although we have range check logic in the host side(which checks the legality of a GM address), the correctness of a GM from vGPU side are supposed to be guaranteed by the drm mm allocator - all those ballooned out spaces are marked as reserved.

Quoted 40 lines

And probably GPU reset should also be disallowed for vGPU instances?

Quoted 37 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +0 -3
      if (!HAS_LLC(dev))
          dev_priv->gtt.base.mm.color_adjust = i915_gtt_color_adjust;
@@ -1702,9 +1713,6 @@ int i915_gem_setup_global_gtt(struct drm_device
 *dev,
          vma->bound |= GLOBAL_BIND;
      }
 
-    dev_priv->gtt.base.start = start;
-    dev_priv->gtt.base.total = end - start;
-
      /* Clear any non-preallocated blocks */
      drm_mm_for_each_hole(entry, &ggtt_vm->mm, hole_start, hole_end) {

DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",

drivers/gpu/drm/i915/i915_gem_gtt.c +3 -0
@@ -1756,6 +1764,9 @@ void i915_global_gtt_cleanup(struct drm_device
 *dev)
      }
 
      if (drm_mm_initialized(&vm->mm)) {
+        if (intel_vgpu_active(dev))
+            intel_vgt_deballoon();
+
          drm_mm_takedown(&vm->mm);
          list_del(&vm->global_link);
      }
drivers/gpu/drm/i915/i915_vgpu.c +3 -0
index 3f6b797..ff5fba3 100644
@@ -83,3 +83,152 @@ void i915_check_vgpu(struct drm_device *dev)
      dev_priv->vgpu.active = true;
      DRM_INFO("Virtual GPU for Intel GVT-g detected.\n");
  }
+
+struct _balloon_info_ {
+    /*
Quoted 8 lines

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 94

Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c +6 -0
+     * There are up to 2 regions per low/high graphic memory that
+     * might be ballooned. Here, index 0/1 is for low
+     * graphic memory, 2/3 for high graphic memory.
+     */
+    struct drm_mm_node space[4];
+} bl_info;

This should be static I think.

Yes, you are right.

Quoted 16 lines

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 106

Quoted 13 lines
drivers/gpu/drm/i915/i915_vgpu.c +4 -0
+/**
+ * intel_vgt_deballoon - deballoon reserved graphics address trunks
+ *
+ * This function is called to deallocate the ballooned-out graphic

memory, when
+ * driver is unloaded or when ballooning fails.
+ */

drivers/gpu/drm/i915/i915_vgpu.c +5 -0
+void intel_vgt_deballoon(void)
+{
+    int i;
+
+    DRM_INFO("VGT deballoon.\n");

Would debug be more appropriate? I don't see much value of saying this on driver unload - it's not that it is optional at this point.

OK. :)

Quoted 3 lines

Also for all logging, is intended human readable name VGT or vGT? If the latter it would be nicer to log it in that form.

Well, I prefer VGT. :)

Quoted 15 lines
Quoted 15 lines
drivers/gpu/drm/i915/i915_vgpu.c +14 -0
+
+    for (i = 0; i < 4; i++) {
+        if (bl_info.space[i].allocated)
+            drm_mm_remove_node(&bl_info.space[i]);
+    }
+
+    memset(&bl_info, 0, sizeof(bl_info));
+}
+
+static int vgt_balloon_space(struct drm_mm *mm,
+                 struct drm_mm_node *node,
+                 unsigned long start, unsigned long end)
+{
+    unsigned long size = end - start;
Quoted 8 lines

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 126

Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c +6 -0
+
+    if (start == end)
+        return -EINVAL;
+
+    DRM_INFO("balloon space: range [ 0x%lx - 0x%lx ] %lu KB.\n",
+         start, end, size / 1024);

KiB ?

Yes. Thanks. :)

Quoted 13 lines
Quoted 13 lines
drivers/gpu/drm/i915/i915_vgpu.c +11 -0
+    node->start = start;
+    node->size = size;
+
+    return drm_mm_reserve_node(mm, node);
+}
+
+/**
+ * intel_vgt_balloon - balloon out reserved graphics address trunks
+ * @dev: drm device
+ *
+ * This function is called at the initialization stage, to balloon

out the

Quoted 8 lines
Quoted 6 lines

+ * graphic address space allocated to other VMs, by marking these spaces as
+ * reserved.
+ *
+ * The ballooning related knowledges(starting address and size of the low/high

s/knowledges\(/knowledge /

Got it.

Quoted 17 lines
Quoted 17 lines

+ * graphic memory) are depicted in the vgt_if structure in a reserved MMIO
+ * range.
+ *
+ * Returns:
+ * zero on success, non-zero if configuration invalid or ballooning failed
+ */

drivers/gpu/drm/i915/i915_vgpu.c +8 -0
+int intel_vgt_balloon(struct drm_device *dev)
+{
+    struct drm_i915_private *dev_priv = to_i915(dev);
+    struct i915_address_space *ggtt_vm = &dev_priv->gtt.base;
+    unsigned long ggtt_vm_end = ggtt_vm->start + ggtt_vm->total;
+
+    unsigned long low_gm_base, low_gm_size, low_gm_end;
+    unsigned long high_gm_base, high_gm_size, high_gm_end;
Quoted 8 lines

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 162

Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c +6 -0
+    int ret;
+
+    low_gm_base = I915_READ(vgtif_reg(avail_rs.low_gmadr.my_base));
+    low_gm_size = I915_READ(vgtif_reg(avail_rs.low_gmadr.my_size));
+    high_gm_base = I915_READ(vgtif_reg(avail_rs.high_gmadr.my_base));
+    high_gm_size = I915_READ(vgtif_reg(avail_rs.high_gmadr.my_size));

Get rid of my_ prefix ?

OK. Will do. :)

Quoted 14 lines
Quoted 14 lines
drivers/gpu/drm/i915/i915_vgpu.c +13 -0
+
+    low_gm_end = low_gm_base + low_gm_size;
+    high_gm_end = high_gm_base + high_gm_size;
+
+    DRM_INFO("VGT ballooning configuration:\n");
+    DRM_INFO("Low graphic memory: base 0x%lx size %ldKB\n",
+         low_gm_base, low_gm_size / 1024);
+    DRM_INFO("High graphic memory: base 0x%lx size %ldKB\n",
+         high_gm_base, high_gm_size / 1024);
+
+    if (low_gm_base < ggtt_vm->start
+        || low_gm_end > dev_priv->gtt.mappable_end
+        || high_gm_base < dev_priv->gtt.mappable_end
Quoted 8 lines

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 181

Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c +6 -0
+        || high_gm_end > ggtt_vm_end) {
+        DRM_ERROR("Invalid ballooning configuration!\n");
+        return -EINVAL;
+    }
+
+    memset(&bl_info, 0, sizeof(bl_info));

If bl_info is static then you don't need this memset?

Oh, yes. :)

Quoted 46 lines
Quoted 34 lines
drivers/gpu/drm/i915/i915_vgpu.c +33 -0
+    /* High graphic memory ballooning */
+    if (high_gm_base > dev_priv->gtt.mappable_end) {
+        ret = vgt_balloon_space(&ggtt_vm->mm,
+                    &bl_info.space[2],
+                    dev_priv->gtt.mappable_end,
+                    high_gm_base);
+
+        if (ret)
+            goto err;
+    }
+
+    /*
+     * No need to partition out the last physical page,
+     * because it is reserved to the guard page.
+     */
+    if (high_gm_end < ggtt_vm_end - PAGE_SIZE) {
+        ret = vgt_balloon_space(&ggtt_vm->mm,
+                    &bl_info.space[3],
+                    high_gm_end,
+                    ggtt_vm_end - PAGE_SIZE);
+        if (ret)
+            goto err;
+    }
+
+    /* Low graphic memory ballooning */
+    if (low_gm_base > ggtt_vm->start) {
+        ret = vgt_balloon_space(&ggtt_vm->mm,
+                    &bl_info.space[0],
+                    ggtt_vm->start, low_gm_base);
+
+        if (ret)
+            goto err;
+    }

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 225

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10 -0
+
+    if (low_gm_end < dev_priv->gtt.mappable_end) {
+        ret = vgt_balloon_space(&ggtt_vm->mm,
+                    &bl_info.space[1],
+                    low_gm_end,
+                    dev_priv->gtt.mappable_end);
+
+        if (ret)
+            goto err;
+    }

Okay, I've figured it out. :) I suppose going back to patch 1, where it

Quoted 8 lines

says "Each VM can only have one portion of continuous area for now", with the emphasis on _one_. That threw me off thinking you have two "balloons" meaning forbidden areas. And then with low and high naming I got the wrong idea that one balloon marks the bottom inaccessible part, and the other top. I didn't figure out the whole low == mappable, high == non-mappable split. I suppose it was my inexperience, but if you can think of a way on how to improve the comment, even ASCII art would be very nice if possible.

Hah. Guess I need a better comments above. Let me think how to organize. :)

Quoted 5 lines

Actually ballooning in the first place confuses me because I thought prior use for that in virtualisation was for memory which can shrink and grow at runtime. Did I get that wrong?

Well, for memory virtualization, the word 'ballooning' does mean this - it can shrink and grow.
Yet for our Intel GVT-g, the 'ballooning' is used, to emphasis that we are not just partitioning the GM space to each vgpu, the vgpu can also perceive its GM address space as starting from zero(although the usable GM spaces may probably not). By now, we do not grow/shrink. :)

Quoted 20 lines
Quoted 20 lines
drivers/gpu/drm/i915/i915_vgpu.c +8 -0
+    DRM_INFO("VGT balloon successfully\n");
+    return 0;
+
+err:
+    DRM_ERROR("VGT balloon fail\n");
+    intel_vgt_deballoon();
+    return ret;
+}
drivers/gpu/drm/i915/i915_vgpu.h +2 -0
index 5f41d01c..f538b18 100644
@@ -81,5 +81,7 @@ struct vgt_if {
      (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)
 
  extern void i915_check_vgpu(struct drm_device *dev);
+extern int intel_vgt_balloon(struct drm_device *dev);
+extern void intel_vgt_deballoon(void);
 
Quoted 8 lines

drivers/gpu/drm/i915/i915_vgpu.h struct vgt_if { · line 87

Quoted 2 lines
drivers/gpu/drm/i915/i915_vgpu.h +0 -0
  #endif /* _I915_VGPU_H_ */

Regards,

Tvrtko

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#13 lore raw

Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic

Quoted 63 lines · Zhang
Quoted 46 lines · Tvrtko Ursulin
Quoted 44 lines · Yu Zhang

With Intel GVT-g, the global graphic memory space is partitioned by multiple vGPU instances in different VMs. The ballooning code is called in i915_gem_setup_global_gtt(), utilizing the drm mm allocator APIs to mark the graphic address space which are partitioned out to other vGPUs as reserved.

v2:
take Chris and Daniel's comments:
- no guard page between different VMs
- use drm_mm_reserve_node() to do the reservation for ballooning, instead of the previous drm_mm_insert_node_in_range_generic()

v3:
take Daniel's comments:
- move ballooning functions into i915_vgpu.c
- add kerneldoc to ballooning functions

---
  drivers/gpu/drm/i915/i915_gem_gtt.c |  17 +++-
  drivers/gpu/drm/i915/i915_vgpu.c    | 149
++++++++++++++++++++++++++++++++++++
  drivers/gpu/drm/i915/i915_vgpu.h    |   2 +
  3 files changed, 165 insertions(+), 3 deletions(-)
drivers/gpu/drm/i915/i915_gem_gtt.c +1 -0
index de12017..2dfac13 100644
@@ -27,6 +27,7 @@
  #include <drm/drmP.h>
  #include <drm/i915_drm.h>
  #include "i915_drv.h"
+#include "i915_vgpu.h"
  #include "i915_trace.h"
  #include "intel_drv.h"
 
@@ -1683,6 +1684,16 @@ int i915_gem_setup_global_gtt(struct drm_device
 *dev,
 
Quoted 16 lines

drivers/gpu/drm/i915/i915_gem_gtt.c int i915_gem_setup_global_gtt(struct drm_device *dev, · line 1696

Quoted 12 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +10 -0
      /* Subtract the guard page ... */
      drm_mm_init(&ggtt_vm->mm, start, end - start - PAGE_SIZE);
+
+    dev_priv->gtt.base.start = start;
+    dev_priv->gtt.base.total = end - start;
+
+    if (intel_vgpu_active(dev)) {
+        ret = intel_vgt_balloon(dev);
+        if (ret)
+            return ret;
+    }
+

Out of curiosity, what will be the mechanism to prevent a vGPU instance from ignoring the ballooning data? Must be something in the hypervisor blocking pass-through access to such domains?

Quoted 2 lines

Well, although we have range check logic in the host side(which checks the legality of a GM address), the correctness of a GM from vGPU side

Oh, I thought that part is direct for performance reasons (avoiding translation). It's not then?

Quoted 2 lines

are supposed to be guaranteed by the drm mm allocator - all those ballooned out spaces are marked as reserved.

I was thinking about a malicious i915 driver instance, or simply a bug in DRM or i915 which breaks the blocked ranges assumption.

If that is possible then it sounds pretty bad if one VM could snoop the surfaces from another.

Quoted 64 lines
Quoted 39 lines

And probably GPU reset should also be disallowed for vGPU instances?

Quoted 37 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +0 -3
      if (!HAS_LLC(dev))
          dev_priv->gtt.base.mm.color_adjust = i915_gtt_color_adjust;
@@ -1702,9 +1713,6 @@ int i915_gem_setup_global_gtt(struct drm_device
 *dev,
          vma->bound |= GLOBAL_BIND;
      }
 
-    dev_priv->gtt.base.start = start;
-    dev_priv->gtt.base.total = end - start;
-
      /* Clear any non-preallocated blocks */
      drm_mm_for_each_hole(entry, &ggtt_vm->mm, hole_start, hole_end) {

DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",

drivers/gpu/drm/i915/i915_gem_gtt.c +3 -0
@@ -1756,6 +1764,9 @@ void i915_global_gtt_cleanup(struct drm_device
 *dev)
      }
 
      if (drm_mm_initialized(&vm->mm)) {
+        if (intel_vgpu_active(dev))
+            intel_vgt_deballoon();
+
          drm_mm_takedown(&vm->mm);
          list_del(&vm->global_link);
      }
drivers/gpu/drm/i915/i915_vgpu.c +3 -0
index 3f6b797..ff5fba3 100644
@@ -83,3 +83,152 @@ void i915_check_vgpu(struct drm_device *dev)
      dev_priv->vgpu.active = true;
      DRM_INFO("Virtual GPU for Intel GVT-g detected.\n");
  }
+
+struct _balloon_info_ {
+    /*
Quoted 8 lines

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 94

Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c +6 -0
+     * There are up to 2 regions per low/high graphic memory that
+     * might be ballooned. Here, index 0/1 is for low
+     * graphic memory, 2/3 for high graphic memory.
+     */
+    struct drm_mm_node space[4];
+} bl_info;

This should be static I think.

Yes, you are right.

Quoted 16 lines

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 106

Quoted 13 lines
drivers/gpu/drm/i915/i915_vgpu.c +4 -0
+/**
+ * intel_vgt_deballoon - deballoon reserved graphics address trunks
+ *
+ * This function is called to deallocate the ballooned-out graphic

memory, when
+ * driver is unloaded or when ballooning fails.
+ */

drivers/gpu/drm/i915/i915_vgpu.c +5 -0
+void intel_vgt_deballoon(void)
+{
+    int i;
+
+    DRM_INFO("VGT deballoon.\n");

Would debug be more appropriate? I don't see much value of saying this on driver unload - it's not that it is optional at this point.

Quoted 5 lines

OK. :)

Quoted 3 lines

Also for all logging, is intended human readable name VGT or vGT? If the latter it would be nicer to log it in that form.

Well, I prefer VGT. :)

So it's vGPU and VGT.. OK... :) (and GVT-d) :)

Quoted 148 lines
Quoted 14 lines
Quoted 14 lines
drivers/gpu/drm/i915/i915_vgpu.c +14 -0
+
+    for (i = 0; i < 4; i++) {
+        if (bl_info.space[i].allocated)
+            drm_mm_remove_node(&bl_info.space[i]);
+    }
+
+    memset(&bl_info, 0, sizeof(bl_info));
+}
+
+static int vgt_balloon_space(struct drm_mm *mm,
+                 struct drm_mm_node *node,
+                 unsigned long start, unsigned long end)
+{
+    unsigned long size = end - start;
Quoted 8 lines

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 126

Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c +6 -0
+
+    if (start == end)
+        return -EINVAL;
+
+    DRM_INFO("balloon space: range [ 0x%lx - 0x%lx ] %lu KB.\n",
+         start, end, size / 1024);

KiB ?

Yes. Thanks. :)

Quoted 13 lines
Quoted 13 lines
drivers/gpu/drm/i915/i915_vgpu.c +11 -0
+    node->start = start;
+    node->size = size;
+
+    return drm_mm_reserve_node(mm, node);
+}
+
+/**
+ * intel_vgt_balloon - balloon out reserved graphics address trunks
+ * @dev: drm device
+ *
+ * This function is called at the initialization stage, to balloon

out the

Quoted 8 lines
Quoted 6 lines

+ * graphic address space allocated to other VMs, by marking these spaces as
+ * reserved.
+ *
+ * The ballooning related knowledges(starting address and size of the low/high

s/knowledges\(/knowledge /

Got it.

Quoted 17 lines
Quoted 17 lines

+ * graphic memory) are depicted in the vgt_if structure in a reserved MMIO
+ * range.
+ *
+ * Returns:
+ * zero on success, non-zero if configuration invalid or ballooning failed
+ */

drivers/gpu/drm/i915/i915_vgpu.c +8 -0
+int intel_vgt_balloon(struct drm_device *dev)
+{
+    struct drm_i915_private *dev_priv = to_i915(dev);
+    struct i915_address_space *ggtt_vm = &dev_priv->gtt.base;
+    unsigned long ggtt_vm_end = ggtt_vm->start + ggtt_vm->total;
+
+    unsigned long low_gm_base, low_gm_size, low_gm_end;
+    unsigned long high_gm_base, high_gm_size, high_gm_end;
Quoted 8 lines

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 162

Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c +6 -0
+    int ret;
+
+    low_gm_base = I915_READ(vgtif_reg(avail_rs.low_gmadr.my_base));
+    low_gm_size = I915_READ(vgtif_reg(avail_rs.low_gmadr.my_size));
+    high_gm_base = I915_READ(vgtif_reg(avail_rs.high_gmadr.my_base));
+    high_gm_size = I915_READ(vgtif_reg(avail_rs.high_gmadr.my_size));

Get rid of my_ prefix ?

OK. Will do. :)

Quoted 14 lines
Quoted 14 lines
drivers/gpu/drm/i915/i915_vgpu.c +13 -0
+
+    low_gm_end = low_gm_base + low_gm_size;
+    high_gm_end = high_gm_base + high_gm_size;
+
+    DRM_INFO("VGT ballooning configuration:\n");
+    DRM_INFO("Low graphic memory: base 0x%lx size %ldKB\n",
+         low_gm_base, low_gm_size / 1024);
+    DRM_INFO("High graphic memory: base 0x%lx size %ldKB\n",
+         high_gm_base, high_gm_size / 1024);
+
+    if (low_gm_base < ggtt_vm->start
+        || low_gm_end > dev_priv->gtt.mappable_end
+        || high_gm_base < dev_priv->gtt.mappable_end
Quoted 8 lines

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 181

Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c +6 -0
+        || high_gm_end > ggtt_vm_end) {
+        DRM_ERROR("Invalid ballooning configuration!\n");
+        return -EINVAL;
+    }
+
+    memset(&bl_info, 0, sizeof(bl_info));

If bl_info is static then you don't need this memset?

Oh, yes. :)

Quoted 46 lines
Quoted 34 lines
drivers/gpu/drm/i915/i915_vgpu.c +33 -0
+    /* High graphic memory ballooning */
+    if (high_gm_base > dev_priv->gtt.mappable_end) {
+        ret = vgt_balloon_space(&ggtt_vm->mm,
+                    &bl_info.space[2],
+                    dev_priv->gtt.mappable_end,
+                    high_gm_base);
+
+        if (ret)
+            goto err;
+    }
+
+    /*
+     * No need to partition out the last physical page,
+     * because it is reserved to the guard page.
+     */
+    if (high_gm_end < ggtt_vm_end - PAGE_SIZE) {
+        ret = vgt_balloon_space(&ggtt_vm->mm,
+                    &bl_info.space[3],
+                    high_gm_end,
+                    ggtt_vm_end - PAGE_SIZE);
+        if (ret)
+            goto err;
+    }
+
+    /* Low graphic memory ballooning */
+    if (low_gm_base > ggtt_vm->start) {
+        ret = vgt_balloon_space(&ggtt_vm->mm,
+                    &bl_info.space[0],
+                    ggtt_vm->start, low_gm_base);
+
+        if (ret)
+            goto err;
+    }

drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 225

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10 -0
+
+    if (low_gm_end < dev_priv->gtt.mappable_end) {
+        ret = vgt_balloon_space(&ggtt_vm->mm,
+                    &bl_info.space[1],
+                    low_gm_end,
+                    dev_priv->gtt.mappable_end);
+
+        if (ret)
+            goto err;
+    }

Okay, I've figured it out. :) I suppose going back to patch 1, where it

Quoted 8 lines

says "Each VM can only have one portion of continuous area for now", with the emphasis on _one_. That threw me off thinking you have two "balloons" meaning forbidden areas. And then with low and high naming I got the wrong idea that one balloon marks the bottom inaccessible part, and the other top. I didn't figure out the whole low == mappable, high == non-mappable split. I suppose it was my inexperience, but if you can think of a way on how to improve the comment, even ASCII art would be very nice if possible.

Quoted 2 lines

Hah. Guess I need a better comments above. Let me think how to organize. :)

Cool thanks!

Quoted 11 lines
Quoted 4 lines

Actually ballooning in the first place confuses me because I thought prior use for that in virtualisation was for memory which can shrink and grow at runtime. Did I get that wrong?

Well, for memory virtualization, the word 'ballooning' does mean this - it can shrink and grow.
Yet for our Intel GVT-g, the 'ballooning' is used, to emphasis that we are not just partitioning the GM space to each vgpu, the vgpu can also perceive its GM address space as starting from zero(although the usable GM spaces may probably not).
By now, we do not grow/shrink. :)

Unless there are plan to make it grow and shrink it seems that we agree it is confusing. But I suppose it is too late to change the term now?

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#14 lore raw

Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic

Hi,

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

Out of curiosity, what will be the mechanism to prevent a vGPU instance from ignoring the ballooning data? Must be something in the hypervisor blocking pass-through access to such domains?

Well, although we have range check logic in the host side(which checks the legality of a GM address), the correctness of a GM from vGPU side

Quoted 8 lines

Oh, I thought that part is direct for performance reasons (avoiding translation). It's not then?

Quoted 2 lines

are supposed to be guaranteed by the drm mm allocator - all those ballooned out spaces are marked as reserved.

I was thinking about a malicious i915 driver instance, or simply a bug in DRM or i915 which breaks the blocked ranges assumption.

Cover letter had a link to a longish paper explaining how all this works in detail.

Short summary:
* Direct access is denied by simply mapping only the guests piece of memory into the guest address space.
* Indirect access (via exec buffer commands) is denied by verifying the exec buffer commands (map read-only + verify + hand over to the hardware when checks pass).
* The ballooning basically makes the guest aware what the offset for its piece of memory has, so the references in the execbuffer don't need to be translated, only verified.

Quoted 2 lines

Unless there are plan to make it grow and shrink it seems that we agree it is confusing. But I suppose it is too late to change the term now?

It's confusing indeed (same question came up @ kvm forum after kvmgt presentation). So a better name would be good I think.

cheers,
Gerd

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#15 lore raw

Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic

Hi,

Quoted 18 lines · Gerd Hoffmann
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

Out of curiosity, what will be the mechanism to prevent a vGPU instance from ignoring the ballooning data? Must be something in the hypervisor blocking pass-through access to such domains?

Well, although we have range check logic in the host side(which checks the legality of a GM address), the correctness of a GM from vGPU side

Quoted 8 lines

Oh, I thought that part is direct for performance reasons (avoiding translation). It's not then?

Quoted 2 lines

are supposed to be guaranteed by the drm mm allocator - all those ballooned out spaces are marked as reserved.

I was thinking about a malicious i915 driver instance, or simply a bug in DRM or i915 which breaks the blocked ranges assumption.

Cover letter had a link to a longish paper explaining how all this works in detail.

Quoted 9 lines

Short summary:
* Direct access is denied by simply mapping only the guests piece of memory into the guest address space.
* Indirect access (via exec buffer commands) is denied by verifying the exec buffer commands (map read-only + verify + hand over to the hardware when checks pass).
* The ballooning basically makes the guest aware what the offset for its piece of memory has, so the references in the execbuffer don't need to be translated, only verified.

Thanks, I did read the blog and powerpoint from the link, but didn't find what you wrote below even after the second read. Probably missed due to lack of domain knowledge.

Anyway, I just wanted to hear that the DRM allocation range is not the only thing preventing access outside the allocated range. :)

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#16 lore raw

Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic

Quoted 33 lines · Tvrtko Ursulin

Hi,

Quoted 19 lines · Gerd Hoffmann
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines

Out of curiosity, what will be the mechanism to prevent a vGPU instance
from ignoring the ballooning data? Must be something in the hypervisor blocking pass-through access to such domains?

Well, although we have range check logic in the host side(which checks the legality of a GM address), the correctness of a GM from vGPU side

Quoted 8 lines

Oh, I thought that part is direct for performance reasons (avoiding translation). It's not then?

Quoted 2 lines

are supposed to be guaranteed by the drm mm allocator - all those ballooned out spaces are marked as reserved.

I was thinking about a malicious i915 driver instance, or simply a bug in DRM or i915 which breaks the blocked ranges assumption.

Cover letter had a link to a longish paper explaining how all this works in detail.

Quoted 9 lines

Short summary:
* Direct access is denied by simply mapping only the guests piece of memory into the guest address space.
* Indirect access (via exec buffer commands) is denied by verifying the exec buffer commands (map read-only + verify + hand over to the hardware when checks pass).
* The ballooning basically makes the guest aware what the offset for its piece of memory has, so the references in the execbuffer don't need to be translated, only verified.

Quoted 6 lines

Thanks, I did read the blog and powerpoint from the link, but didn't find what you wrote below even after the second read. Probably missed due to lack of domain knowledge.

Anyway, I just wanted to hear that the DRM allocation range is not the only thing preventing access outside the allocated range. :)

Thanks, Tvrtko & Gerd.
DRM allocation range is not the only guarantee. We also have range check functions in our host code. MMIO accesses and ring buffer commands will be checked/scanned in the host side to see if there's any illegal GM address.

Quoted 6 lines

Regards,

Tvrtko

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#17 lore raw

Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic

Quoted 12 lines

From: Yu, Zhang
Sent: Wednesday, December 17, 2014 11:10 AM

Quoted 10 lines · Gerd Hoffmann
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

Out of curiosity, what will be the mechanism to prevent a vGPU instance
from ignoring the ballooning data? Must be something in the hypervisor blocking pass-through access to such domains?

Well, although we have range check logic in the host side(which checks the legality of a GM address), the correctness of a GM from vGPU side

Oh, I thought that part is direct for performance reasons (avoiding translation). It's not then?

yes, just for performance reason to avoid translation

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

are supposed to be guaranteed by the drm mm allocator - all those ballooned out spaces are marked as reserved.

I was thinking about a malicious i915 driver instance, or simply a bug in DRM or i915 which breaks the blocked ranges assumption.

host will validate all the addresses submitted by the guest, w/ the assumption that ballooning is in place so if the address is outside of the allocated machine graphics memory address, it's caught and prevented. We use ballooning to let guest handle the correctness itself, but having Xen to ensure isolation.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Cover letter had a link to a longish paper explaining how all this works in detail.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Short summary:
* Direct access is denied by simply mapping only the guests piece of memory into the guest address space.
* Indirect access (via exec buffer commands) is denied by verifying the exec buffer commands (map read-only + verify + hand over to the hardware when checks pass).
* The ballooning basically makes the guest aware what the offset for its piece of memory has, so the references in the execbuffer don't need to be translated, only verified.

correct.

Quoted 5 lines
Quoted 5 lines

Thanks, I did read the blog and powerpoint from the link, but didn't find what you wrote below even after the second read. Probably missed due to lack of domain knowledge.

Quoted 8 lines
Quoted 2 lines

Anyway, I just wanted to hear that the DRM allocation range is not the only thing preventing access outside the allocated range. :)

Thanks, Tvrtko & Gerd.
DRM allocation range is not the only guarantee. We also have range check functions in our host code. MMIO accesses and ring buffer commands will be checked/scanned in the host side to see if there's any illegal GM address.

and if you read the paper referred in 1st mail, we call our ballooning as address space ballooning which is different from traditional ballooning technique used in memory virtualization, w/ the major difference as:

* traditional ballooning only care about the size. As long as the desired number of pages can be ballooned out, we don't care whether they actually come from.

* address space ballooning used in our case care about both size and address. We need the guest to balloon out a specified range of addresses as explained earlier.

whether to grow/shrink dynamic is not the key difference between two approaches. yes, we only support static ballooning now at boot time, but the same interface can be extended to support dynamic ballooning in the future, w/ more cooperation from guest driver in the future.

So I think using ballooning is a right fit here, though the policy is very simple now.

Thanks
Kevin
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#18 lore raw

Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic

Quoted 12 lines · Kevin

and if you read the paper referred in 1st mail, we call our ballooning as address space ballooning which is different from traditional ballooning technique used in memory virtualization, w/ the major difference as:

* traditional ballooning only care about the size. As long as the desired number of pages can be ballooned out, we don't care whether they actually come from.

* address space ballooning used in our case care about both size and address. We need the guest to balloon out a specified range of addresses as explained earlier.

Quoted 7 lines

whether to grow/shrink dynamic is not the key difference between two approaches. yes, we only support static ballooning now at boot time, but the same interface can be extended to support dynamic ballooning in the future, w/ more cooperation from guest driver in the future.

So I think using ballooning is a right fit here, though the policy is very simple now.

Personally I think the first part of the argument does not hold since ballooning is the key word here - address space or memory makes little difference to the metaphor. It is more like partitioning of blanking as it stands.

You only get away with it since you hint of plans to make it dynamic in the future. :)

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#19 lore raw

Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic

In reply to Tvrtko Ursulin
Quoted 22 lines

From: Tvrtko Ursulin
Sent: Friday, December 12, 2014 9:00 PM

Review int i915_gem_setup_global_gtt(struct

Quoted 1 line
(quoted diff) +0 -0
@@ -1683,6 +1684,16 @@ int i915_gem_setup_global_gtt(struct

drm_device *dev,

drivers/gpu/drm/i915/i915_gem_gtt.c int i915_gem_setup_global_gtt(struct drm_device *dev, · line 1696

Quoted 13 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +10 -0
  	/* Subtract the guard page ... */
  	drm_mm_init(&ggtt_vm->mm, start, end - start - PAGE_SIZE);
+
+	dev_priv->gtt.base.start = start;
+	dev_priv->gtt.base.total = end - start;
+
+	if (intel_vgpu_active(dev)) {
+		ret = intel_vgt_balloon(dev);
+		if (ret)
+			return ret;
+	}
+

Out of curiosity, what will be the mechanism to prevent a vGPU instance from ignoring the ballooning data? Must be something in the hypervisor blocking pass-through access to such domains?

Quoted 2 lines

And probably GPU reset should also be disallowed for vGPU instances?

GPU reset is allowed and the host side needs to emulate the vGPU reset behavior. A simple example is that we need remove this vGPU from the scheduling queue, upon a vGPU reset request.

Thanks
Kevin
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#20 lore raw

PATCH v3 3/8 drm/i915: Partition the fence registers for vGPU in i915 driver

In reply to Yu Zhang

With Intel GVT-g, the fence registers are partitioned by multiple vGPU instances in different VMs. Routine i915_gem_load() is modified to reset the num_fence_regs, when the driver detects it's running in a VM. And the allocated fence number is provided in PV INFO page structure.

---
 drivers/gpu/drm/i915/i915_gem.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +5 -0
index 1de94cc..0c8b32e 100644
@@ -29,6 +29,7 @@
 #include <drm/drm_vma_manager.h>
 #include <drm/i915_drm.h>
 #include "i915_drv.h"
+#include "i915_vgpu.h"
 #include "i915_trace.h"
 #include "intel_drv.h"
 #include <linux/oom.h>
@@ -5014,6 +5015,10 @@ i915_gem_load(struct drm_device *dev)
 	else
 		dev_priv->num_fence_regs = 8;
 
+	if (intel_vgpu_active(dev))
+		dev_priv->num_fence_regs =
+				I915_READ(vgtif_reg(avail_rs.fence_num));
+
 	/* Initialize fence registers to zero */
 	INIT_LIST_HEAD(&dev_priv->mm.fence_list);
 	i915_gem_restore_fences(dev);
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#21 lore raw

Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

Quoted 27 lines · Yu Zhang

With Intel GVT-g, the fence registers are partitioned by multiple vGPU instances in different VMs. Routine i915_gem_load() is modified to reset the num_fence_regs, when the driver detects it's running in a VM. And the allocated fence number is provided in PV INFO page structure.

---
  drivers/gpu/drm/i915/i915_gem.c | 5 +++++
  1 file changed, 5 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +1 -0
index 1de94cc..0c8b32e 100644
@@ -29,6 +29,7 @@
  #include <drm/drm_vma_manager.h>
  #include <drm/i915_drm.h>
  #include "i915_drv.h"
+#include "i915_vgpu.h"
  #include "i915_trace.h"
  #include "intel_drv.h"
  #include <linux/oom.h>

drivers/gpu/drm/i915/i915_gem.c i915_gem_load(struct drm_device *dev) · line 5024

Quoted 10 lines
drivers/gpu/drm/i915/i915_gem.c +4 -0
@@ -5014,6 +5015,10 @@ i915_gem_load(struct drm_device *dev)
  	else
  		dev_priv->num_fence_regs = 8;
 
+	if (intel_vgpu_active(dev))
+		dev_priv->num_fence_regs =
+				I915_READ(vgtif_reg(avail_rs.fence_num));
+
  	/* Initialize fence registers to zero */
  	INIT_LIST_HEAD(&dev_priv->mm.fence_list);
  	i915_gem_restore_fences(dev);

You don't need a start offset and number of allocated fences per domain?

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#22 lore raw

Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

Quoted 33 lines · Tvrtko Ursulin
Quoted 30 lines · Yu Zhang

With Intel GVT-g, the fence registers are partitioned by multiple vGPU instances in different VMs. Routine i915_gem_load() is modified to reset the num_fence_regs, when the driver detects it's running in a VM. And the allocated fence number is provided in PV INFO page structure.

---
  drivers/gpu/drm/i915/i915_gem.c | 5 +++++
  1 file changed, 5 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +1 -0
index 1de94cc..0c8b32e 100644
@@ -29,6 +29,7 @@
  #include <drm/drm_vma_manager.h>
  #include <drm/i915_drm.h>
  #include "i915_drv.h"
+#include "i915_vgpu.h"
  #include "i915_trace.h"
  #include "intel_drv.h"
  #include <linux/oom.h>
@@ -5014,6 +5015,10 @@ i915_gem_load(struct drm_device *dev)
      else
          dev_priv->num_fence_regs = 8;
 
Quoted 10 lines

drivers/gpu/drm/i915/i915_gem.c i915_gem_load(struct drm_device *dev) · line 5024

Quoted 8 lines
drivers/gpu/drm/i915/i915_gem.c +4 -0
+    if (intel_vgpu_active(dev))
+        dev_priv->num_fence_regs =
+                I915_READ(vgtif_reg(avail_rs.fence_num));
+
      /* Initialize fence registers to zero */
      INIT_LIST_HEAD(&dev_priv->mm.fence_list);
      i915_gem_restore_fences(dev);

You don't need a start offset and number of allocated fences per domain?

The PV INFO structure is shared between each vgpu and host, so I guess this is per domain? Not sure if I get your exact meaning. :)

Quoted 3 lines

Regards,

B.R.
Yu

Quoted 3 lines

Tvrtko

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#23 lore raw

Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

Quoted 42 lines · Zhang
Quoted 31 lines · Tvrtko Ursulin
Quoted 30 lines · Yu Zhang

With Intel GVT-g, the fence registers are partitioned by multiple vGPU instances in different VMs. Routine i915_gem_load() is modified to reset the num_fence_regs, when the driver detects it's running in a VM. And the allocated fence number is provided in PV INFO page structure.

---
  drivers/gpu/drm/i915/i915_gem.c | 5 +++++
  1 file changed, 5 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +1 -0
index 1de94cc..0c8b32e 100644
@@ -29,6 +29,7 @@
  #include <drm/drm_vma_manager.h>
  #include <drm/i915_drm.h>
  #include "i915_drv.h"
+#include "i915_vgpu.h"
  #include "i915_trace.h"
  #include "intel_drv.h"
  #include <linux/oom.h>
@@ -5014,6 +5015,10 @@ i915_gem_load(struct drm_device *dev)
      else
          dev_priv->num_fence_regs = 8;
 
Quoted 10 lines

drivers/gpu/drm/i915/i915_gem.c i915_gem_load(struct drm_device *dev) · line 5024

Quoted 8 lines
drivers/gpu/drm/i915/i915_gem.c +4 -0
+    if (intel_vgpu_active(dev))
+        dev_priv->num_fence_regs =
+                I915_READ(vgtif_reg(avail_rs.fence_num));
+
      /* Initialize fence registers to zero */
      INIT_LIST_HEAD(&dev_priv->mm.fence_list);
      i915_gem_restore_fences(dev);

You don't need a start offset and number of allocated fences per domain?

Quoted 3 lines

The PV INFO structure is shared between each vgpu and host, so I guess this is per domain?
Not sure if I get your exact meaning. :)

I didn't figure out how each domain knowns which fences to use? They know how many, but which ones?

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#24 lore raw

Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

Hi,

Quoted 2 lines

I didn't figure out how each domain knowns which fences to use? They know how many, but which ones?

I think the guest doesn't have to know because mmio access is trapped by the hypervisor anyway.

cheers,
Gerd

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#25 lore raw

Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

Hi,

Quoted 5 lines · Gerd Hoffmann
Quoted 2 lines

I didn't figure out how each domain knowns which fences to use? They know how many, but which ones?

I think the guest doesn't have to know because mmio access is trapped by the hypervisor anyway.

Okay.. so the point is to ensure no domain can hog all fences, but if the hypervisor is already tracking everything why is this limiting required then in the guest? Hypervisor could just say all fences are allocated since it knows how much it gave out per guest.

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#26 lore raw

Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

Quoted 6 lines · Tvrtko Ursulin

Hi,

Quoted 3 lines · Gerd Hoffmann
Quoted 2 lines

I didn't figure out how each domain knowns which fences to use? They know how many, but which ones?

Quoted 7 lines
Quoted 2 lines

I think the guest doesn't have to know because mmio access is trapped by the hypervisor anyway.

Okay.. so the point is to ensure no domain can hog all fences, but if the hypervisor is already tracking everything why is this limiting required then in the guest? Hypervisor could just say all fences are allocated since it knows how much it gave out per guest.

There is no dynamic allocation. Each guest gets a fixed number of fences assigned which it can use, pretty much like memory where each guest gets a fixed piece too.

It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.

cheers,
Gerd

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#27 lore raw

Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

Quoted 19 lines · Gerd Hoffmann
Quoted 6 lines · Tvrtko Ursulin

Hi,

Quoted 3 lines · Gerd Hoffmann
Quoted 2 lines

I didn't figure out how each domain knowns which fences to use? They know how many, but which ones?

Quoted 7 lines
Quoted 2 lines

I think the guest doesn't have to know because mmio access is trapped by the hypervisor anyway.

Okay.. so the point is to ensure no domain can hog all fences, but if the hypervisor is already tracking everything why is this limiting required then in the guest? Hypervisor could just say all fences are allocated since it knows how much it gave out per guest.

There is no dynamic allocation. Each guest gets a fixed number of fences assigned which it can use, pretty much like memory where each guest gets a fixed piece too.

Quoted 5 lines

It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.

That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how many?

With this scheme it could be one guest wants more and can't get them even if no one else is using any. If not limited they could be dynamically allocated by the hypervisor, no?

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#28 lore raw

Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

Hi,

Quoted 8 lines
Quoted 5 lines

It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.

That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how many?

There is a fixed assignment of fences to guests, so it's a fixed number. But as the hypervisor is involved in any fence access anyway there is no need for the guest to know which of the fences it owns, the hypervisor can remap that transparently for the guest, without performance penalty.

Quoted 3 lines

With this scheme it could be one guest wants more and can't get them even if no one else is using any. If not limited they could be dynamically allocated by the hypervisor, no?

Should be possible as extension, but I think[1] for now the goal is to stay as close as possible to physical hardware, where you can't dynamically allocate fences in the first place.

cheers,
Gerd

[1] Not being deeply involved into vgpu development, just reviewing the bits with some kvm background.

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#29 lore raw

Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

Quoted 8 lines · Gerd Hoffmann
   Hi,
Quoted 6 lines
Quoted 5 lines

It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.

Quoted 7 lines
Quoted 2 lines

That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how many?

There is a fixed assignment of fences to guests, so it's a fixed number. But as the hypervisor is involved in any fence access anyway there is no need for the guest to know which of the fences it owns, the hypervisor can remap that transparently for the guest, without performance penalty.

Thanks Gerd. Exactly.
Although fence registers are parititioned to vGPU, it is not necessary for a vGPU to know the physical mmio addresses of the allocated fence registers.
For example, vGPU 1 with fence size 4 can access the fence registers from 0x100000-10001f; at the same time, vGPU 2 with fence size 8 can access the fence registers from 0x100000-0x10003f. Although this seems conflicting, it does not matter. Because these mmio addresses are all supposed to be trapped in the host side, which will keep a record of the real fence offset of different vGPUs(say 0 for vGPU 1 and 4 for vGPU 2), and then does the remapping. Therefore, the physical operations on the fence register will be performed by host code on different ones(say, 0x100000-10001fh for vGPU 1 and 0x100020-0x10005f for vGPU 2).

Quoted 8 lines
Quoted 4 lines

With this scheme it could be one guest wants more and can't get them even if no one else is using any. If not limited they could be dynamically allocated by the hypervisor, no?

Should be possible as extension, but I think[1] for now the goal is to stay as close as possible to physical hardware, where you can't dynamically allocate fences in the first place.

Yes. By now we have feature to configure the fence numbers for a vGPU. But we do not support the dynamical allocations.

Quoted 10 lines

cheers,
Gerd

[1] Not being deeply involved into vgpu development, just reviewing the bits with some kvm background.

Thanks
Yu
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#30 lore raw

Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

Quoted 16 lines · Zhang
Quoted 8 lines · Gerd Hoffmann
   Hi,
Quoted 6 lines
Quoted 5 lines

It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.

Quoted 7 lines
Quoted 2 lines

That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how many?

There is a fixed assignment of fences to guests, so it's a fixed number. But as the hypervisor is involved in any fence access anyway there is no need for the guest to know which of the fences it owns, the hypervisor can remap that transparently for the guest, without performance penalty.

Quoted 13 lines

Thanks Gerd. Exactly.
Although fence registers are parititioned to vGPU, it is not necessary for a vGPU to know the physical mmio addresses of the allocated fence registers.
For example, vGPU 1 with fence size 4 can access the fence registers from 0x100000-10001f; at the same time, vGPU 2 with fence size 8 can access the fence registers from 0x100000-0x10003f. Although this seems conflicting, it does not matter. Because these mmio addresses are all supposed to be trapped in the host side, which will keep a record of the real fence offset of different vGPUs(say 0 for vGPU 1 and 4 for vGPU 2), and then does the remapping. Therefore, the physical operations on the fence register will be performed by host code on different ones(say, 0x100000-10001fh for vGPU 1 and 0x100020-0x10005f for vGPU 2).

Okay, I think I get it now. What I had in mind is not really possible without a dedicated hypervisor<->guest communication channel. Or in other words you would have to extend the way i915 allocates them from mmio writes to something bi-directional.

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#31 lore raw

Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

Quoted 32 lines · Tvrtko Ursulin
Quoted 16 lines · Zhang
Quoted 8 lines · Gerd Hoffmann

Hi,

Quoted 6 lines
Quoted 5 lines

It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.

Quoted 7 lines
Quoted 2 lines

That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how many?

There is a fixed assignment of fences to guests, so it's a fixed number. But as the hypervisor is involved in any fence access anyway there is no need for the guest to know which of the fences it owns, the hypervisor can remap that transparently for the guest, without performance penalty.

Quoted 13 lines

Thanks Gerd. Exactly.
Although fence registers are parititioned to vGPU, it is not necessary for a vGPU to know the physical mmio addresses of the allocated fence registers.
For example, vGPU 1 with fence size 4 can access the fence registers from 0x100000-10001f; at the same time, vGPU 2 with fence size 8 can access the fence registers from 0x100000-0x10003f. Although this seems conflicting, it does not matter. Because these mmio addresses are all supposed to be trapped in the host side, which will keep a record of the real fence offset of different vGPUs(say 0 for vGPU 1 and 4 for vGPU 2), and then does the remapping. Therefore, the physical operations on the fence register will be performed by host code on different ones(say, 0x100000-10001fh for vGPU 1 and 0x100020-0x10005f for vGPU 2).

Quoted 4 lines

Okay, I think I get it now. What I had in mind is not really possible without a dedicated hypervisor<->guest communication channel. Or in other words you would have to extend the way i915 allocates them from mmio writes to something bi-directional.

You could virtualize fences the same way we virtualize fences for userspace for gtt mmap access: If we need to steal a fences we simply need to unmap the relevant gtt mmio range from the guest ptes. This should work well since on current platforms the only thing that really needs fences is cpu access, the gpu doesn't need them. Well except for some oddball cases in the display block, but those are virtualized anyway (not fbc for guests or anything else like that).

This would also fit a bit more closely with how the host manages fences, so benefiting the new kvm/xengt-on-i915 mode for the host instead of the current implementation which also virtualizes host i915 access cycles. -Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#32 lore raw

Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

Quoted 46 lines · Daniel Vetter
Quoted 32 lines · Tvrtko Ursulin
Quoted 16 lines · Zhang
Quoted 8 lines · Gerd Hoffmann

Hi,

Quoted 6 lines
Quoted 5 lines

It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.

Quoted 7 lines
Quoted 2 lines

That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how many?

There is a fixed assignment of fences to guests, so it's a fixed number. But as the hypervisor is involved in any fence access anyway there is no need for the guest to know which of the fences it owns, the hypervisor can remap that transparently for the guest, without performance penalty.

Quoted 13 lines

Thanks Gerd. Exactly.
Although fence registers are parititioned to vGPU, it is not necessary for a vGPU to know the physical mmio addresses of the allocated fence registers.
For example, vGPU 1 with fence size 4 can access the fence registers from 0x100000-10001f; at the same time, vGPU 2 with fence size 8 can access the fence registers from 0x100000-0x10003f. Although this seems conflicting, it does not matter. Because these mmio addresses are all supposed to be trapped in the host side, which will keep a record of the real fence offset of different vGPUs(say 0 for vGPU 1 and 4 for vGPU 2), and then does the remapping. Therefore, the physical operations on the fence register will be performed by host code on different ones(say, 0x100000-10001fh for vGPU 1 and 0x100020-0x10005f for vGPU 2).

Quoted 4 lines

Okay, I think I get it now. What I had in mind is not really possible without a dedicated hypervisor<->guest communication channel. Or in other words you would have to extend the way i915 allocates them from mmio writes to something bi-directional.

You could virtualize fences the same way we virtualize fences for userspace for gtt mmap access: If we need to steal a fences we simply need to unmap the relevant gtt mmio range from the guest ptes. This should work well since on current platforms the only thing that really needs fences is cpu access, the gpu doesn't need them. Well except for some oddball cases in the display block, but those are virtualized anyway (not fbc for guests or anything else like that).

Quoted 3 lines

This would also fit a bit more closely with how the host manages fences, so benefiting the new kvm/xengt-on-i915 mode for the host instead of the current implementation which also virtualizes host i915 access cycles.

Btw this isn't a blocker for the current implementation since we can always just tell guests that they can use all fences when we implement this. So the current, simpler implementation isn't restricting us in any meaningful way.
-Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#33 lore raw

Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

In reply to Daniel Vetter
Quoted 42 lines

From: Daniel Vetter
Sent: Thursday, December 18, 2014 1:10 AM

Quoted 32 lines · Tvrtko Ursulin
Quoted 16 lines · Zhang
Quoted 8 lines · Gerd Hoffmann

Hi,

Quoted 6 lines
Quoted 5 lines

It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.

Quoted 7 lines
Quoted 2 lines

That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how many?

There is a fixed assignment of fences to guests, so it's a fixed number. But as the hypervisor is involved in any fence access anyway there is no need for the guest to know which of the fences it owns, the hypervisor can remap that transparently for the guest, without performance penalty.

Quoted 13 lines

Thanks Gerd. Exactly.
Although fence registers are parititioned to vGPU, it is not necessary for a vGPU to know the physical mmio addresses of the allocated fence registers.
For example, vGPU 1 with fence size 4 can access the fence registers from 0x100000-10001f; at the same time, vGPU 2 with fence size 8 can access the fence registers from 0x100000-0x10003f. Although this seems conflicting, it does not matter. Because these mmio addresses are all supposed to be trapped in the host side, which will keep a record of the real fence offset of different vGPUs(say 0 for vGPU 1 and 4 for vGPU 2), and then does the remapping. Therefore, the physical operations on the fence register will be performed by host code on different ones(say, 0x100000-10001fh for vGPU 1 and 0x100020-0x10005f for vGPU 2).

Quoted 3 lines

Okay, I think I get it now. What I had in mind is not really possible without a dedicated hypervisor<->guest communication channel. Or in other words you would have to extend the way i915 allocates them from mmio

writes

Quoted 1 line

to something bi-directional.

Quoted 7 lines

You could virtualize fences the same way we virtualize fences for userspace for gtt mmap access: If we need to steal a fences we simply need to unmap the relevant gtt mmio range from the guest ptes. This should work well since on current platforms the only thing that really needs fences is cpu access, the gpu doesn't need them. Well except for some oddball cases in the display block, but those are virtualized anyway (not fbc for guests or anything else like that).

doing this w/o guest's awareness is challenging. bear in mind that vCPU scheduling (in hypervisor) is split from vGPU scheduling (in xengt driver). so xengt driver doesn't know when cpu access happens, and thus steal a fence under the hood is problematic.

based on current enlightenment framework, it's possible to have driver allocate/release fences through PVINFO window. That way we can avoid the current static partitioning scheme, and the changes shouldn't be that much.

btw curious I heard that typically 2-4 fences are OK for most workloads. Is that true? If yes, this fence partition along is not the only limiting factor on #VM. the 512M aperture size is more limiting (but will be better starting from bdw)

Quoted 5 lines

This would also fit a bit more closely with how the host manages fences, so benefiting the new kvm/xengt-on-i915 mode for the host instead of the current implementation which also virtualizes host i915 access cycles. -Daniel

for host yes we won't virtualize fence then. We're evaluating how to remove most mediations on the host paths, with few exceptions e.g. for registers which requires manual save/restore at context swtich time.

Thanks
Kevin
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#34 lore raw

Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

Quoted 50 lines · Kevin
Quoted 42 lines

From: Daniel Vetter
Sent: Thursday, December 18, 2014 1:10 AM

Quoted 32 lines · Tvrtko Ursulin
Quoted 16 lines · Zhang
Quoted 8 lines · Gerd Hoffmann

Hi,

Quoted 6 lines
Quoted 5 lines

It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.

Quoted 7 lines
Quoted 2 lines

That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how many?

There is a fixed assignment of fences to guests, so it's a fixed number. But as the hypervisor is involved in any fence access anyway there is no need for the guest to know which of the fences it owns, the hypervisor can remap that transparently for the guest, without performance penalty.

Quoted 13 lines

Thanks Gerd. Exactly.
Although fence registers are parititioned to vGPU, it is not necessary for a vGPU to know the physical mmio addresses of the allocated fence registers.
For example, vGPU 1 with fence size 4 can access the fence registers from 0x100000-10001f; at the same time, vGPU 2 with fence size 8 can access the fence registers from 0x100000-0x10003f. Although this seems conflicting, it does not matter. Because these mmio addresses are all supposed to be trapped in the host side, which will keep a record of the real fence offset of different vGPUs(say 0 for vGPU 1 and 4 for vGPU 2), and then does the remapping. Therefore, the physical operations on the fence register will be performed by host code on different ones(say, 0x100000-10001fh for vGPU 1 and 0x100020-0x10005f for vGPU 2).

Quoted 3 lines

Okay, I think I get it now. What I had in mind is not really possible without a dedicated hypervisor<->guest communication channel. Or in other words you would have to extend the way i915 allocates them from mmio

writes

Quoted 1 line

to something bi-directional.

Quoted 7 lines

You could virtualize fences the same way we virtualize fences for userspace for gtt mmap access: If we need to steal a fences we simply need to unmap the relevant gtt mmio range from the guest ptes. This should work well since on current platforms the only thing that really needs fences is cpu access, the gpu doesn't need them. Well except for some oddball cases in the display block, but those are virtualized anyway (not fbc for guests or anything else like that).

Quoted 4 lines

doing this w/o guest's awareness is challenging. bear in mind that vCPU scheduling (in hypervisor) is split from vGPU scheduling (in xengt driver). so xengt driver doesn't know when cpu access happens, and thus steal a fence under the hood is problematic.

Well i915 also doesn't interfere with the linux scheduler at all to manage fences. All you have to do is kick out the ptes before yanking the fence away, no need to talk to the scheduler at all. Ofc if some other vm is using that area it'll immediately fault on the cpu, but the fault handler can then rectify this and steal some other fence that's not used right away again hopefully.

Quoted 16 lines

based on current enlightenment framework, it's possible to have driver allocate/release fences through PVINFO window. That way we can avoid the current static partitioning scheme, and the changes shouldn't be that much.

btw curious I heard that typically 2-4 fences are OK for most workloads. Is that true? If yes, this fence partition along is not the only limiting factor on #VM. the 512M aperture size is more limiting (but will be better starting from bdw)

Quoted 5 lines

This would also fit a bit more closely with how the host manages fences, so benefiting the new kvm/xengt-on-i915 mode for the host instead of the current implementation which also virtualizes host i915 access cycles. -Daniel

Quoted 4 lines

for host yes we won't virtualize fence then. We're evaluating how to remove most mediations on the host paths, with few exceptions e.g. for registers which requires manual save/restore at context swtich time.

My idea would be to use additional MI_LRI/SRM commands before/after the call to load the context into the ring using the functions in i915_gem_context.c. This is exactly what the execbuf code does too:
There's lots of additional stuff (per-platform) it does around the actual context switch. So even there I think you can implement the VM switch on top of i915 infrastructure. Maybe we need to duplicate/export some execbuf internals, but that's a lot better than duplicating the entire driver and having 2 drivers fight over the same hw.
-Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#35 lore raw

Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

Quoted 67 lines

From: Daniel Vetter [mailto:[email protected]] On Behalf Of Daniel Vetter
Sent: Thursday, December 18, 2014 4:08 PM

Quoted 11 lines · Kevin
Quoted 11 lines

From: Daniel Vetter
Sent: Thursday, December 18, 2014 1:10 AM

Quoted 7 lines · Tvrtko Ursulin
Quoted 5 lines · Zhang
Quoted 4 lines · Gerd Hoffmann

Hi,

Quoted 2 lines
Quoted 2 lines

It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the

hypervisor

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.

That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how

many?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

There is a fixed assignment of fences to guests, so it's a fixed number. But as the hypervisor is involved in any fence access anyway there is

no

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

need for the guest to know which of the fences it owns, the hypervisor can remap that transparently for the guest, without performance

penalty.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Thanks Gerd. Exactly.
Although fence registers are parititioned to vGPU, it is not necessary for a vGPU to know the physical mmio addresses of the allocated fence registers.
For example, vGPU 1 with fence size 4 can access the fence registers from 0x100000-10001f; at the same time, vGPU 2 with fence size 8 can access the fence registers from 0x100000-0x10003f. Although this

seems

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

conflicting, it does not matter. Because these mmio addresses are all supposed to be trapped in the host side, which will keep a record of the real fence offset of different vGPUs(say 0 for vGPU 1 and 4 for vGPU 2), and then does the remapping. Therefore, the physical operations on the fence register will be performed by host code on different ones(say, 0x100000-10001fh for vGPU 1 and 0x100020-0x10005f for vGPU 2).

Okay, I think I get it now. What I had in mind is not really possible without a dedicated hypervisor<->guest communication channel. Or in

other

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

words you would have to extend the way i915 allocates them from mmio

writes

Quoted 1 line

to something bi-directional.

You could virtualize fences the same way we virtualize fences for userspace for gtt mmap access: If we need to steal a fences we simply

need

Quoted 1 line
Quoted 1 line

to unmap the relevant gtt mmio range from the guest ptes. This should

work

Quoted 9 lines
Quoted 4 lines

well since on current platforms the only thing that really needs fences is cpu access, the gpu doesn't need them. Well except for some oddball cases in the display block, but those are virtualized anyway (not fbc for guests or anything else like that).

doing this w/o guest's awareness is challenging. bear in mind that vCPU scheduling (in hypervisor) is split from vGPU scheduling (in xengt driver). so xengt driver doesn't know when cpu access happens, and thus steal a fence under the hood is problematic.

Quoted 6 lines

Well i915 also doesn't interfere with the linux scheduler at all to manage fences. All you have to do is kick out the ptes before yanking the fence away, no need to talk to the scheduler at all. Ofc if some other vm is using that area it'll immediately fault on the cpu, but the fault handler can then rectify this and steal some other fence that's not used right away again hopefully.

yes, this would work, but I'd think having guest to allocate/release fence through PVINFO is simpler and less hypervisor specific.

Quoted 22 lines
Quoted 17 lines

based on current enlightenment framework, it's possible to have driver allocate/release fences through PVINFO window. That way we can avoid the current static partitioning scheme, and the changes shouldn't be that much.

btw curious I heard that typically 2-4 fences are OK for most workloads. Is that true? If yes, this fence partition along is not the only limiting factor on #VM. the 512M aperture size is more limiting (but will be better starting from bdw)

Quoted 5 lines

This would also fit a bit more closely with how the host manages fences, so benefiting the new kvm/xengt-on-i915 mode for the host instead of the current implementation which also virtualizes host i915 access cycles. -Daniel

Quoted 4 lines

for host yes we won't virtualize fence then. We're evaluating how to remove most mediations on the host paths, with few exceptions e.g. for registers which requires manual save/restore at context swtich time.

Quoted 10 lines

My idea would be to use additional MI_LRI/SRM commands before/after the call to load the context into the ring using the functions in i915_gem_context.c. This is exactly what the execbuf code does too:
There's lots of additional stuff (per-platform) it does around the actual context switch. So even there I think you can implement the VM switch on top of i915 infrastructure. Maybe we need to duplicate/export some execbuf internals, but that's a lot better than duplicating the entire driver and having 2 drivers fight over the same hw.
-Daniel
--

yes, that's the direction what we agreed before and definitely we don't want to duplicate entire driver (btw even with current i915-on-xengt model our driver code is much smaller since we only focus on resource management. :-)) We'll try to leverage existing i915 functions as many as possible. For major changes, we'll summarize and send out for design review to reach agreement. for above example, yes what you described should work, by hooking xengt specific register list in 915_gem_context.c

Thanks
Kevin
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#36 lore raw

Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver

In reply to Tvrtko Ursulin
Quoted 47 lines

From: Tvrtko Ursulin
Sent: Tuesday, December 16, 2014 9:44 PM

Quoted 42 lines · Zhang
Quoted 41 lines · Tvrtko Ursulin
Quoted 28 lines · Yu Zhang

With Intel GVT-g, the fence registers are partitioned by multiple vGPU instances in different VMs. Routine i915_gem_load() is modified to reset the num_fence_regs, when the driver detects it's running in a VM. And the allocated fence number is provided in PV INFO page structure.

---
  drivers/gpu/drm/i915/i915_gem.c | 5 +++++
  1 file changed, 5 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +1 -0
index 1de94cc..0c8b32e 100644
@@ -29,6 +29,7 @@
  #include <drm/drm_vma_manager.h>
  #include <drm/i915_drm.h>
  #include "i915_drv.h"
+#include "i915_vgpu.h"
  #include "i915_trace.h"
  #include "intel_drv.h"
  #include <linux/oom.h>

drivers/gpu/drm/i915/i915_gem.c i915_gem_load(struct drm_device *dev) · line 5024

Quoted 10 lines
drivers/gpu/drm/i915/i915_gem.c +4 -0
@@ -5014,6 +5015,10 @@ i915_gem_load(struct drm_device *dev)
      else
          dev_priv->num_fence_regs = 8;
 
+    if (intel_vgpu_active(dev))
+        dev_priv->num_fence_regs =
+                I915_READ(vgtif_reg(avail_rs.fence_num));
+
      /* Initialize fence registers to zero */
      INIT_LIST_HEAD(&dev_priv->mm.fence_list);
      i915_gem_restore_fences(dev);

You don't need a start offset and number of allocated fences per domain?

Quoted 7 lines
Quoted 3 lines

The PV INFO structure is shared between each vgpu and host, so I guess this is per domain?
Not sure if I get your exact meaning. :)

I didn't figure out how each domain knowns which fences to use? They know how many, but which ones?

the offset is maintained in the host side, which will be added to guest index in relevant emulation path.

Thanks
Kevin
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#37 lore raw

PATCH v3 4/8 drm/i915: Disable framebuffer compression for i915 driver in VM

In reply to Yu Zhang

Framebuffer compression is disabled when driver detects it's running in a Intel GVT-g enlightened VM, because FBC is not emulated and there is no stolen memory for a vGPU.

v2:
take Chris' comments:
- move the code into intel_update_fbc()

---
 drivers/gpu/drm/i915/intel_pm.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/intel_pm.c +4 -0
index 7a69eba..3bc5d93 100644
@@ -544,6 +544,10 @@ void intel_update_fbc(struct drm_device *dev)
 		return;
 	}
 
+	/* disable framebuffer compression in vGPU */
+	if (intel_vgpu_active(dev))
+		i915.enable_fbc = 0;
+
 	/*
 	 * If FBC is already on, we just have to verify that we can
 	 * keep it that way...
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#38 lore raw

Re: [PATCH v3 4/8] drm/i915: Disable framebuffer compression for i915 driver in VM

Quoted 21 lines · Yu Zhang

Framebuffer compression is disabled when driver detects it's running in a Intel GVT-g enlightened VM, because FBC is not emulated and there is no stolen memory for a vGPU.

v2:
take Chris' comments:
- move the code into intel_update_fbc()

---
  drivers/gpu/drm/i915/intel_pm.c | 4 ++++
  1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/intel_pm.c +0 -0
index 7a69eba..3bc5d93 100644

drivers/gpu/drm/i915/intel_pm.c void intel_update_fbc(struct drm_device *dev) · line 553

Quoted 10 lines
drivers/gpu/drm/i915/intel_pm.c +4 -0
@@ -544,6 +544,10 @@ void intel_update_fbc(struct drm_device *dev)
  		return;
  	}
 
+	/* disable framebuffer compression in vGPU */
+	if (intel_vgpu_active(dev))
+		i915.enable_fbc = 0;
+
  	/*
  	 * If FBC is already on, we just have to verify that we can
  	 * keep it that way...

Looks like you'll need to rebase this one, I see no intel_update_fbc in my tree. :(

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#39 lore raw

Re: [PATCH v3 4/8] drm/i915: Disable framebuffer compression for i915 driver in VM

Quoted 27 lines · Tvrtko Ursulin
Quoted 24 lines · Yu Zhang

Framebuffer compression is disabled when driver detects it's running in a Intel GVT-g enlightened VM, because FBC is not emulated and there is no stolen memory for a vGPU.

v2:
take Chris' comments:
- move the code into intel_update_fbc()

---
  drivers/gpu/drm/i915/intel_pm.c | 4 ++++
  1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/intel_pm.c +0 -0
index 7a69eba..3bc5d93 100644
@@ -544,6 +544,10 @@ void intel_update_fbc(struct drm_device *dev)
          return;
      }
 
Quoted 12 lines

drivers/gpu/drm/i915/intel_pm.c void intel_update_fbc(struct drm_device *dev) · line 553

Quoted 8 lines
drivers/gpu/drm/i915/intel_pm.c +4 -0
+    /* disable framebuffer compression in vGPU */
+    if (intel_vgpu_active(dev))
+        i915.enable_fbc = 0;
+
      /*
       * If FBC is already on, we just have to verify that we can
       * keep it that way...

Looks like you'll need to rebase this one, I see no intel_update_fbc in my tree. :(

OK. I'll rebase this in next version. :)

Quoted 5 lines

Regards,

Tvrtko

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#40 lore raw

PATCH v3 5/8 drm/i915: Add the display switch logic for vGPU in i915 driver

In reply to Yu Zhang

Display switch logic is added to notify the host side that current vGPU have a valid surface to show. It does so by writing the display_ready field in PV INFO page, and then will be handled in the host side. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host side.

v2:
- move the notification code outside the 'else' in load sequence
- remove the notification code in intel_crtc_set_config()

---
 drivers/gpu/drm/i915/i915_dma.c  | 11 +++++++++++
 drivers/gpu/drm/i915/i915_vgpu.h |  7 +++++++
 2 files changed, 18 insertions(+)
drivers/gpu/drm/i915/i915_dma.c +11 -0
index 9a73533..06daa5c 100644
@@ -36,6 +36,7 @@
 #include "intel_drv.h"
 #include <drm/i915_drm.h>
 #include "i915_drv.h"
+#include "i915_vgpu.h"
 #include "i915_trace.h"
 #include <linux/pci.h>
 #include <linux/console.h>
@@ -1780,6 +1781,16 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags)
 		dev_priv->ums.mm_suspended = 1;
 	}
 
+	if (intel_vgpu_active(dev)) {
+		/*
+		 * Notify a valid surface after modesetting,
+		 * when running inside a VM.
+		 */
+		struct drm_i915_private *dev_priv = to_i915(dev);
+
+		I915_WRITE(vgtif_reg(display_ready), VGT_DRV_DISPLAY_READY);
+	}
+
 	i915_setup_sysfs(dev);
 
 	if (INTEL_INFO(dev)->num_pipes) {
drivers/gpu/drm/i915/i915_vgpu.h +7 -0
index f538b18..9d35fb8 100644
@@ -80,6 +80,13 @@ struct vgt_if {
 #define vgtif_reg(x) \
 	(VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)
 
+/* vGPU display status to be used by the host side */
+enum vgt_display_status {
+	VGT_DRV_DISPLAY_NOT_READY = 0,
+	VGT_DRV_DISPLAY_READY,  /* ready for display switch */
+};
+
+
 extern void i915_check_vgpu(struct drm_device *dev);
 extern int intel_vgt_balloon(struct drm_device *dev);
 extern void intel_vgt_deballoon(void);
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Reviewed-by on v3 5/8
#41 lore raw

Re: [PATCH v3 5/8] drm/i915: Add the display switch logic for vGPU in i915 driver

Quoted 36 lines · Yu Zhang

Display switch logic is added to notify the host side that current vGPU have a valid surface to show. It does so by writing the display_ready field in PV INFO page, and then will be handled in the host side. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host side.

v2:
- move the notification code outside the 'else' in load sequence
- remove the notification code in intel_crtc_set_config()

---
  drivers/gpu/drm/i915/i915_dma.c  | 11 +++++++++++
  drivers/gpu/drm/i915/i915_vgpu.h |  7 +++++++
  2 files changed, 18 insertions(+)
drivers/gpu/drm/i915/i915_dma.c +1 -0
index 9a73533..06daa5c 100644
@@ -36,6 +36,7 @@
  #include "intel_drv.h"
  #include <drm/i915_drm.h>
  #include "i915_drv.h"
+#include "i915_vgpu.h"
  #include "i915_trace.h"
  #include <linux/pci.h>
  #include <linux/console.h>
@@ -1780,6 +1781,16 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags)
  		dev_priv->ums.mm_suspended = 1;
  	}
 

drivers/gpu/drm/i915/i915_dma.c int i915_driver_load(struct drm_device *dev, unsigned long flags) · line 1793

Quoted 10 lines
drivers/gpu/drm/i915/i915_dma.c +10 -0
+	if (intel_vgpu_active(dev)) {
+		/*
+		 * Notify a valid surface after modesetting,
+		 * when running inside a VM.
+		 */
+		struct drm_i915_private *dev_priv = to_i915(dev);
+
+		I915_WRITE(vgtif_reg(display_ready), VGT_DRV_DISPLAY_READY);
+	}
+

Current code base already has dev_priv so looks like another rebase. Otherwise pretty simple so:

Quoted 12 lines
drivers/gpu/drm/i915/i915_dma.c +0 -0
  	i915_setup_sysfs(dev);
 
  	if (INTEL_INFO(dev)->num_pipes) {
drivers/gpu/drm/i915/i915_vgpu.h +0 -0
index f538b18..9d35fb8 100644
@@ -80,6 +80,13 @@ struct vgt_if {
  #define vgtif_reg(x) \
  	(VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)
 

drivers/gpu/drm/i915/i915_vgpu.h struct vgt_if { · line 92

Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h +7 -0
+/* vGPU display status to be used by the host side */
+enum vgt_display_status {
+	VGT_DRV_DISPLAY_NOT_READY = 0,
+	VGT_DRV_DISPLAY_READY,  /* ready for display switch */
+};
+
+
  extern void i915_check_vgpu(struct drm_device *dev);
  extern int intel_vgt_balloon(struct drm_device *dev);
  extern void intel_vgt_deballoon(void);

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#42 lore raw

Re: [PATCH v3 5/8] drm/i915: Add the display switch logic for vGPU in i915 driver

In reply to Yu Zhang
Quoted 54 lines · Yu Zhang

Display switch logic is added to notify the host side that current vGPU have a valid surface to show. It does so by writing the display_ready field in PV INFO page, and then will be handled in the host side. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host side.

v2:
- move the notification code outside the 'else' in load sequence
- remove the notification code in intel_crtc_set_config()

---
 drivers/gpu/drm/i915/i915_dma.c  | 11 +++++++++++
 drivers/gpu/drm/i915/i915_vgpu.h |  7 +++++++
 2 files changed, 18 insertions(+)
drivers/gpu/drm/i915/i915_dma.c +11 -0
index 9a73533..06daa5c 100644
@@ -36,6 +36,7 @@
 #include "intel_drv.h"
 #include <drm/i915_drm.h>
 #include "i915_drv.h"
+#include "i915_vgpu.h"
 #include "i915_trace.h"
 #include <linux/pci.h>
 #include <linux/console.h>
@@ -1780,6 +1781,16 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags)
 		dev_priv->ums.mm_suspended = 1;
 	}
 
+	if (intel_vgpu_active(dev)) {
+		/*
+		 * Notify a valid surface after modesetting,
+		 * when running inside a VM.
+		 */
+		struct drm_i915_private *dev_priv = to_i915(dev);
+
+		I915_WRITE(vgtif_reg(display_ready), VGT_DRV_DISPLAY_READY);
+	}
+
 	i915_setup_sysfs(dev);
 
 	if (INTEL_INFO(dev)->num_pipes) {
drivers/gpu/drm/i915/i915_vgpu.h +0 -0
index f538b18..9d35fb8 100644

drivers/gpu/drm/i915/i915_vgpu.h struct vgt_if { · line 87

Quoted 8 lines
drivers/gpu/drm/i915/i915_vgpu.h +5 -0
@@ -80,6 +80,13 @@ struct vgt_if {
 #define vgtif_reg(x) \
 	(VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)
 
+/* vGPU display status to be used by the host side */
+enum vgt_display_status {
+	VGT_DRV_DISPLAY_NOT_READY = 0,
+	VGT_DRV_DISPLAY_READY,  /* ready for display switch */
+};

Nitpick: Since this is a virtual register I prefer we use plain #defines with explicit codes for everything instead of enums. Makes it clearer which values are used and all that.
-Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#43 lore raw

Re: [PATCH v3 5/8] drm/i915: Add the display switch logic for vGPU in i915 driver

Quoted 58 lines · Daniel Vetter
Quoted 57 lines · Yu Zhang

Display switch logic is added to notify the host side that current vGPU have a valid surface to show. It does so by writing the display_ready field in PV INFO page, and then will be handled in the host side. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host side.

v2:
- move the notification code outside the 'else' in load sequence
- remove the notification code in intel_crtc_set_config()

---
  drivers/gpu/drm/i915/i915_dma.c  | 11 +++++++++++
  drivers/gpu/drm/i915/i915_vgpu.h |  7 +++++++
  2 files changed, 18 insertions(+)
drivers/gpu/drm/i915/i915_dma.c +11 -0
index 9a73533..06daa5c 100644
@@ -36,6 +36,7 @@
  #include "intel_drv.h"
  #include <drm/i915_drm.h>
  #include "i915_drv.h"
+#include "i915_vgpu.h"
  #include "i915_trace.h"
  #include <linux/pci.h>
  #include <linux/console.h>
@@ -1780,6 +1781,16 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags)
  		dev_priv->ums.mm_suspended = 1;
  	}
 
+	if (intel_vgpu_active(dev)) {
+		/*
+		 * Notify a valid surface after modesetting,
+		 * when running inside a VM.
+		 */
+		struct drm_i915_private *dev_priv = to_i915(dev);
+
+		I915_WRITE(vgtif_reg(display_ready), VGT_DRV_DISPLAY_READY);
+	}
+
  	i915_setup_sysfs(dev);
 
  	if (INTEL_INFO(dev)->num_pipes) {
drivers/gpu/drm/i915/i915_vgpu.h +0 -0
index f538b18..9d35fb8 100644
@@ -80,6 +80,13 @@ struct vgt_if {
  #define vgtif_reg(x) \
  	(VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)
 
Quoted 10 lines

drivers/gpu/drm/i915/i915_vgpu.h struct vgt_if { · line 87

Quoted 5 lines
drivers/gpu/drm/i915/i915_vgpu.h +5 -0
+/* vGPU display status to be used by the host side */
+enum vgt_display_status {
+	VGT_DRV_DISPLAY_NOT_READY = 0,
+	VGT_DRV_DISPLAY_READY,  /* ready for display switch */
+};

Nitpick: Since this is a virtual register I prefer we use plain #defines with explicit codes for everything instead of enums. Makes it clearer which values are used and all that.
-Daniel

Got it. #define will be used. And I will rebase this patch.

Quoted 1 line

Thanks
Yu
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#44 lore raw

PATCH v3 6/8 drm/i915: Disable power management for i915 driver in VM

In reply to Yu Zhang

With Intel GVT-g, GPU power management is controlled by host driver, so there is no need to provide virtualized GPU PM support. In the future it might be useful to gather VM input for freq boost, but now let's disable it simply.

v2:
take Chris' comments:
- do not special case this to gen6+

---
 drivers/gpu/drm/i915/intel_pm.c | 4 ++++
 1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/intel_pm.c +4 -0
index 3bc5d93..3722bd4 100644
@@ -5314,6 +5314,10 @@ void intel_enable_gt_powersave(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = dev->dev_private;
 
+	/* Powersaving is controlled by the host when inside a VM */
+	if (intel_vgpu_active(dev))
+		return;
+
 	if (IS_IRONLAKE_M(dev)) {
 		mutex_lock(&dev->struct_mutex);
 		ironlake_enable_drps(dev);
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Reviewed-by on v3 6/8
#45 lore raw

Re: [PATCH v3 6/8] drm/i915: Disable power management for i915 driver in VM

Quoted 21 lines · Yu Zhang

With Intel GVT-g, GPU power management is controlled by host driver, so there is no need to provide virtualized GPU PM support. In the future it might be useful to gather VM input for freq boost, but now let's disable it simply.

v2:
take Chris' comments:
- do not special case this to gen6+

---
  drivers/gpu/drm/i915/intel_pm.c | 4 ++++
  1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/intel_pm.c +0 -0
index 3bc5d93..3722bd4 100644

drivers/gpu/drm/i915/intel_pm.c void intel_enable_gt_powersave(struct drm_device *dev) · line 5323

Quoted 10 lines
drivers/gpu/drm/i915/intel_pm.c +4 -0
@@ -5314,6 +5314,10 @@ void intel_enable_gt_powersave(struct drm_device *dev)
  {
  	struct drm_i915_private *dev_priv = dev->dev_private;
 
+	/* Powersaving is controlled by the host when inside a VM */
+	if (intel_vgpu_active(dev))
+		return;
+
  	if (IS_IRONLAKE_M(dev)) {
  		mutex_lock(&dev->struct_mutex);
  		ironlake_enable_drps(dev);

I was looking for potential call sites of this which come earlier than i915_check_vgpu. Didn't find any but found RPS (intel_pm_setup) - what's with that - should it be disabled as well?

Otherwise,

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#46 lore raw

Re: [PATCH v3 6/8] drm/i915: Disable power management for i915 driver in VM

Quoted 28 lines · Tvrtko Ursulin
Quoted 25 lines · Yu Zhang

With Intel GVT-g, GPU power management is controlled by host driver, so there is no need to provide virtualized GPU PM support. In the future it might be useful to gather VM input for freq boost, but now let's disable it simply.

v2:
take Chris' comments:
- do not special case this to gen6+

---
  drivers/gpu/drm/i915/intel_pm.c | 4 ++++
  1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/intel_pm.c +0 -0
index 3bc5d93..3722bd4 100644
@@ -5314,6 +5314,10 @@ void intel_enable_gt_powersave(struct
 drm_device *dev)
  {
      struct drm_i915_private *dev_priv = dev->dev_private;
 
Quoted 13 lines
Quoted 8 lines
drivers/gpu/drm/i915/intel_pm.c +4 -0
+    /* Powersaving is controlled by the host when inside a VM */
+    if (intel_vgpu_active(dev))
+        return;
+
      if (IS_IRONLAKE_M(dev)) {
          mutex_lock(&dev->struct_mutex);
          ironlake_enable_drps(dev);

I was looking for potential call sites of this which come earlier than i915_check_vgpu. Didn't find any but found RPS (intel_pm_setup) - what's with that - should it be disabled as well?

IIUC, the intel_pm_setup() only add the intel_gen6_powersave_work() to the delayed_resume_work queue. Real call of this routine will be delayed to intel_enable_gt_powersave(). So I guess no necessary to do anything in intel_pm_setup(). :)

Quoted 4 lines

Otherwise,

B.R.
Yu
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#47 lore raw

PATCH v3 7/8 drm/i915: Create vGPU specific write MMIO to reduce traps

In reply to Yu Zhang

In the virtualized environment, forcewake operations are not necessory for the driver, because mmio accesses will be trapped and emulated by the host side, and real forcewake operations are also done in the host. New mmio write handlers are added to directly call the __raw_i915_write, therefore will reduce many traps and increase the overall performance for drivers running in the VM with Intel GVT-g enhancement.

v2:
take Chris' comments:
- register the mmio hooks in intel_uncore_init()
v3:
take Daniel's comments:
- use macros to assign mmio write functions for vGPU

---
 drivers/gpu/drm/i915/intel_uncore.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
drivers/gpu/drm/i915/intel_uncore.c +16 -0
index cae27bb..b76c21d 100644
@@ -727,6 +727,14 @@ hsw_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace)
 	REG_WRITE_FOOTER; \
 }
 
+#define __vgpu_write(x) \
+static void \
+vgpu_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \
+	REG_WRITE_HEADER; \
+	__raw_i915_write##x(dev_priv, reg, val); \
+	REG_WRITE_FOOTER; \
+}
+
 static const u32 gen8_shadowed_regs[] = {
 	FORCEWAKE_MT,
 	GEN6_RPNSWREQ,
@@ -821,6 +829,10 @@ __gen4_write(8)
 __gen4_write(16)
 __gen4_write(32)
 __gen4_write(64)
+__vgpu_write(8)
+__vgpu_write(16)
+__vgpu_write(32)
+__vgpu_write(64)
 
 #undef __chv_write
 #undef __gen8_write
@@ -828,6 +840,7 @@ __gen4_write(64)
 #undef __gen6_write
 #undef __gen5_write
 #undef __gen4_write
+#undef __vgpu_write
 #undef REG_WRITE_FOOTER
 #undef REG_WRITE_HEADER
 
@@ -939,6 +952,9 @@ void intel_uncore_init(struct drm_device *dev)
 		break;
 	}
 
+	if (intel_vgpu_active(dev))
+		ASSIGN_WRITE_MMIO_VFUNCS(vgpu);
+
 	i915_check_and_clear_faults(dev);
 }
 #undef ASSIGN_WRITE_MMIO_VFUNCS
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#48 lore raw

Re: [PATCH v3 7/8] drm/i915: Create vGPU specific write MMIO to reduce traps

Quoted 2 lines · Yu Zhang

In the virtualized environment, forcewake operations are not necessory for the driver, because mmio accesses will be trapped

necessary

Quoted 60 lines

and emulated by the host side, and real forcewake operations are also done in the host. New mmio write handlers are added to directly call the __raw_i915_write, therefore will reduce many traps and increase the overall performance for drivers running in the VM with Intel GVT-g enhancement.

v2:
take Chris' comments:
- register the mmio hooks in intel_uncore_init()
v3:
take Daniel's comments:
- use macros to assign mmio write functions for vGPU

---
  drivers/gpu/drm/i915/intel_uncore.c | 16 ++++++++++++++++
  1 file changed, 16 insertions(+)
drivers/gpu/drm/i915/intel_uncore.c +13 -0
index cae27bb..b76c21d 100644
@@ -727,6 +727,14 @@ hsw_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace)
  	REG_WRITE_FOOTER; \
  }
 
+#define __vgpu_write(x) \
+static void \
+vgpu_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \
+	REG_WRITE_HEADER; \
+	__raw_i915_write##x(dev_priv, reg, val); \
+	REG_WRITE_FOOTER; \
+}
+
  static const u32 gen8_shadowed_regs[] = {
  	FORCEWAKE_MT,
  	GEN6_RPNSWREQ,
@@ -821,6 +829,10 @@ __gen4_write(8)
  __gen4_write(16)
  __gen4_write(32)
  __gen4_write(64)
+__vgpu_write(8)
+__vgpu_write(16)
+__vgpu_write(32)
+__vgpu_write(64)
 
  #undef __chv_write
  #undef __gen8_write
@@ -828,6 +840,7 @@ __gen4_write(64)
  #undef __gen6_write
  #undef __gen5_write
  #undef __gen4_write
+#undef __vgpu_write
  #undef REG_WRITE_FOOTER
  #undef REG_WRITE_HEADER
 

drivers/gpu/drm/i915/intel_uncore.c void intel_uncore_init(struct drm_device *dev) · line 960

Quoted 9 lines
drivers/gpu/drm/i915/intel_uncore.c +3 -0
@@ -939,6 +952,9 @@ void intel_uncore_init(struct drm_device *dev)
  		break;
  	}
 
+	if (intel_vgpu_active(dev))
+		ASSIGN_WRITE_MMIO_VFUNCS(vgpu);
+
  	i915_check_and_clear_faults(dev);
  }
  #undef ASSIGN_WRITE_MMIO_VFUNCS

Maybe I am missing something obvious, but why are read variants not needed?

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#49 lore raw

Re: [PATCH v3 7/8] drm/i915: Create vGPU specific write MMIO to reduce traps

Quoted 6 lines · Tvrtko Ursulin
Quoted 2 lines · Yu Zhang

In the virtualized environment, forcewake operations are not necessory for the driver, because mmio accesses will be trapped

necessary

Thanks.

Quoted 66 lines
Quoted 66 lines

and emulated by the host side, and real forcewake operations are also done in the host. New mmio write handlers are added to directly call the __raw_i915_write, therefore will reduce many traps and increase the overall performance for drivers running in the VM with Intel GVT-g enhancement.

v2:
take Chris' comments:
- register the mmio hooks in intel_uncore_init()
v3:
take Daniel's comments:
- use macros to assign mmio write functions for vGPU

---
  drivers/gpu/drm/i915/intel_uncore.c | 16 ++++++++++++++++
  1 file changed, 16 insertions(+)
drivers/gpu/drm/i915/intel_uncore.c +8 -0
index cae27bb..b76c21d 100644
@@ -727,6 +727,14 @@ hsw_write##x(struct drm_i915_private *dev_priv,
 off_t reg, u##x val, bool trace)
      REG_WRITE_FOOTER; \
  }
 
+#define __vgpu_write(x) \
+static void \
+vgpu_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val,
 bool trace) { \
+    REG_WRITE_HEADER; \
+    __raw_i915_write##x(dev_priv, reg, val); \
+    REG_WRITE_FOOTER; \
+}
+
  static const u32 gen8_shadowed_regs[] = {

FORCEWAKE_MT,
GEN6_RPNSWREQ,

drivers/gpu/drm/i915/intel_uncore.c +5 -0
@@ -821,6 +829,10 @@ __gen4_write(8)
  __gen4_write(16)
  __gen4_write(32)
  __gen4_write(64)
+__vgpu_write(8)
+__vgpu_write(16)
+__vgpu_write(32)
+__vgpu_write(64)
 
  #undef __chv_write
  #undef __gen8_write
@@ -828,6 +840,7 @@ __gen4_write(64)
  #undef __gen6_write
  #undef __gen5_write
  #undef __gen4_write
+#undef __vgpu_write
  #undef REG_WRITE_FOOTER
  #undef REG_WRITE_HEADER
 
@@ -939,6 +952,9 @@ void intel_uncore_init(struct drm_device *dev)
          break;
      }
 
Quoted 9 lines

drivers/gpu/drm/i915/intel_uncore.c void intel_uncore_init(struct drm_device *dev) · line 960

Quoted 7 lines
drivers/gpu/drm/i915/intel_uncore.c +3 -0
+    if (intel_vgpu_active(dev))
+        ASSIGN_WRITE_MMIO_VFUNCS(vgpu);
+
      i915_check_and_clear_faults(dev);
  }
  #undef ASSIGN_WRITE_MMIO_VFUNCS

Maybe I am missing something obvious, but why are read variants not needed?

Wah. You got me. Guess I missed this during rebase. Thank you, Tvrtko.

Quoted 6 lines

Regards,

Tvrtko

B.R.
Yu

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#50 lore raw

PATCH v3 8/8 drm/i915: Support alias ppgtt in VM if ppgtt is enabled

In reply to Yu Zhang

The current Intel GVT-g only supports alias ppgtt. And the emulation is done in the host by first trapping PP_DIR_BASE mmio accesses. Updating PP_DIR_BASE by using instructions such as MI_LOAD_REGISTER_IMM are hard to detect and are not supported in current code. Therefore this patch also added a new callback routine - vgpu_mm_switch() to set the PP_DIR_BASE by mmio writes.

v2:
take Chris' comments:
- move the code into sanitize_enable_ppgtt()

---
 drivers/gpu/drm/i915/i915_gem_gtt.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
drivers/gpu/drm/i915/i915_gem_gtt.c +16 -0
index 2dfac13..55307eb 100644
@@ -44,6 +44,9 @@ static int sanitize_enable_ppgtt(struct drm_device *dev, int enable_ppgtt)
 	if (IS_GEN8(dev))
 		has_full_ppgtt = false; /* XXX why? */
 
+	if (intel_vgpu_active(dev))
+		has_full_ppgtt = false; /* emulation is too hard */
+
 	if (enable_ppgtt == 0 || !has_aliasing_ppgtt)
 		return 0;
 
@@ -733,6 +736,16 @@ static int hsw_mm_switch(struct i915_hw_ppgtt *ppgtt,
 	return 0;
 }
 
+static int vgpu_mm_switch(struct i915_hw_ppgtt *ppgtt,
+			 struct intel_engine_cs *ring)
+{
+	struct drm_i915_private *dev_priv = to_i915(ppgtt->base.dev);
+
+	I915_WRITE(RING_PP_DIR_DCLV(ring), PP_DIR_DCLV_2G);
+	I915_WRITE(RING_PP_DIR_BASE(ring), get_pd_offset(ppgtt));
+	return 0;
+}
+
 static int gen7_mm_switch(struct i915_hw_ppgtt *ppgtt,
 			  struct intel_engine_cs *ring)
 {
@@ -1059,6 +1072,9 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
 	} else
 		BUG();
 
+	if (intel_vgpu_active(dev))
+		ppgtt->switch_mm = vgpu_mm_switch;
+
 	ret = gen6_ppgtt_alloc(ppgtt);
 	if (ret)
 		return ret;
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Tested-by on v3 8/8
#51 lore raw

Re: [PATCH v3 8/8] drm/i915: Support alias ppgtt in VM if

-------------------------------------Summary-------------------------------------
Platform: baseline_drm_intel_nightly_pass_rate->patch_applied_pass_rate
BYT: pass/total=291/291->290/291
PNV: pass/total=356/356->356/356
ILK: pass/total=372/372->371/372
IVB: pass/total=544/546->544/546
SNB: pass/total=423/425->424/425
HSW: pass/total=579/579->579/579
BDW: pass/total=421/423->422/423
-------------------------------------Detailed-------------------------------------
test_platform: test_suite, test_case, result_with_drm_intel_nightly(count, machine_id...)...->result_with_patch_applied(count, machine_id)...
BYT: Intel_gpu_tools, igt_kms_setmode_invalid-clone-single-crtc, TIMEOUT(15, M36M31)PASS(1, M38) -> TIMEOUT(4, M31)
ILK: Intel_gpu_tools, igt_kms_setmode_invalid-clone-single-crtc, FAIL(3, M26)DMESG_FAIL(1, M26)TIMEOUT(24, M37M6M26)PASS(1, M26) -> TIMEOUT(4, M37)
IVB: Intel_gpu_tools, igt_gem_bad_reloc_negative-reloc, NSPT(5, M4M21M34M45)PASS(8, M21M34M4M45) -> NSPT(1, M34)PASS(3, M34)
IVB: Intel_gpu_tools, igt_kms_pipe_crc_basic_hang-read-crc-pipe-B, PASS(4, M4M34) -> TIMEOUT(1, M34)PASS(3, M34)
SNB: Intel_gpu_tools, igt_gem_concurrent_blit_gpuX-bcs-overwrite-source-forked, FAIL(1, M22)PASS(9, M35M22) -> FAIL(1, M22)PASS(3, M22)
BDW: Intel_gpu_tools, igt_gem_reset_stats_ban-bsd, DMESG_WARN(1, M30)PASS(9, M30M42) -> PASS(4, M42)
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

Reviewed-by on v3 8/8
#52 lore raw

Re: [PATCH v3 8/8] drm/i915: Support alias ppgtt in VM if ppgtt is enabled

In reply to Yu Zhang
Quoted 50 lines · Yu Zhang

The current Intel GVT-g only supports alias ppgtt. And the emulation is done in the host by first trapping PP_DIR_BASE mmio accesses. Updating PP_DIR_BASE by using instructions such as MI_LOAD_REGISTER_IMM are hard to detect and are not supported in current code. Therefore this patch also added a new callback routine - vgpu_mm_switch() to set the PP_DIR_BASE by mmio writes.

v2:
take Chris' comments:
- move the code into sanitize_enable_ppgtt()

---
  drivers/gpu/drm/i915/i915_gem_gtt.c | 16 ++++++++++++++++
  1 file changed, 16 insertions(+)
drivers/gpu/drm/i915/i915_gem_gtt.c +13 -0
index 2dfac13..55307eb 100644
@@ -44,6 +44,9 @@ static int sanitize_enable_ppgtt(struct drm_device *dev, int enable_ppgtt)
  	if (IS_GEN8(dev))
  		has_full_ppgtt = false; /* XXX why? */
 
+	if (intel_vgpu_active(dev))
+		has_full_ppgtt = false; /* emulation is too hard */
+
  	if (enable_ppgtt == 0 || !has_aliasing_ppgtt)
  		return 0;
 
@@ -733,6 +736,16 @@ static int hsw_mm_switch(struct i915_hw_ppgtt *ppgtt,
  	return 0;
  }
 
+static int vgpu_mm_switch(struct i915_hw_ppgtt *ppgtt,
+			 struct intel_engine_cs *ring)
+{
+	struct drm_i915_private *dev_priv = to_i915(ppgtt->base.dev);
+
+	I915_WRITE(RING_PP_DIR_DCLV(ring), PP_DIR_DCLV_2G);
+	I915_WRITE(RING_PP_DIR_BASE(ring), get_pd_offset(ppgtt));
+	return 0;
+}
+
  static int gen7_mm_switch(struct i915_hw_ppgtt *ppgtt,
  			  struct intel_engine_cs *ring)
  {

drivers/gpu/drm/i915/i915_gem_gtt.c static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt) · line 1080

Quoted 9 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +3 -0
@@ -1059,6 +1072,9 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt)
  	} else
  		BUG();
 
+	if (intel_vgpu_active(dev))
+		ppgtt->switch_mm = vgpu_mm_switch;
+
  	ret = gen6_ppgtt_alloc(ppgtt);
  	if (ret)
  		return ret;

Pending potential "checkpatch.pl --strict" fixes,

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#53 lore raw
In reply to Yu Zhang
Quoted 45 lines · Yu Zhang

Intel GVT-g (previously known as XenGT), is a complete GPU virtualization solution with mediated pass-through for 4th generation Intel Core processors - Haswell platform. This technology presents a virtual full-fledged GPU to each Virtual Machine (VM). VMs can directly access performance-critical resources, without intervention from the hypervisor in most cases, while privileged operations from VMs are trap-and-emulated at minimal cost. For detail, please refer to
https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-release

This patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, will be sent out in another patchset.

The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory w/o hypervisor's intervention, e.g. filling textures or queuing commands. However w/ the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses w/o complexity of translation.

Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (w/ some faulting mechanism).

The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource XenGT also partitions them among VMs.

Quoted 3 lines

Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.

I think this looks good now overall, so please find someone from the i915 kernel team to review the details. You might need to poke managers to make that happen ;-)
-Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#54 lore raw
Quoted 46 lines · Daniel Vetter
Quoted 45 lines · Yu Zhang

Intel GVT-g (previously known as XenGT), is a complete GPU virtualization solution with mediated pass-through for 4th generation Intel Core processors - Haswell platform. This technology presents a virtual full-fledged GPU to each Virtual Machine (VM). VMs can directly access performance-critical resources, without intervention from the hypervisor in most cases, while privileged operations from VMs are trap-and-emulated at minimal cost. For detail, please refer to
https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-release

This patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, will be sent out in another patchset.

The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory w/o hypervisor's intervention, e.g. filling textures or queuing commands. However w/ the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses w/o complexity of translation.

Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (w/ some faulting mechanism).

The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource XenGT also partitions them among VMs.

Quoted 8 lines
Quoted 3 lines

Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.

I think this looks good now overall, so please find someone from the i915 kernel team to review the details. You might need to poke managers to make that happen ;-)
-Daniel

Thanks a lot, Daniel.
This is great. :-)

Yu

Quoted 1 line

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#55 lore raw
In reply to Yu Zhang

Hi,

I'll try to do the detailed review of your series in the following few days. I might ask some questions on the design also to help me understand the bigger picture.

First thing, I see that patches are checkpatch.pl clean, apart when run in strict mode. I think Daniel prefers "--strict" nowadays, at least I needed to fix up those in my patches so you should probably do the same.

On 11/13/2014 12:02 PM, Yu Zhang wrote:
[snip]

Quoted 7 lines

The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory w/o hypervisor's intervention, e.g. filling textures or queuing commands. However w/ the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT

Quoted 8 lines

core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses w/o complexity of translation.

I couldn't figure this out - is there any hardware protection in there, or a virtual i915 instance could access memory outside it's area if there was a security bug/exploit in the driver, or the balloon was incorrectly set up?

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#56 lore raw
Quoted 10 lines · Tvrtko Ursulin

Hi,

I'll try to do the detailed review of your series in the following few days. I might ask some questions on the design also to help me understand the bigger picture.

First thing, I see that patches are checkpatch.pl clean, apart when run in strict mode. I think Daniel prefers "--strict" nowadays, at least I needed to fix up those in my patches so you should probably do the same.

Well I mostly run --strict since that checks for the alignment of continuation lines for function parameters. Imo not following the i915 coding style for that looks a bit too jarring ;-)

Still checkpatch is just a tool and occasionally seriously lacks taste, so please use --strict wisely instead of strictly.
-Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
maintainer
#57 lore raw
Quoted 6 lines · Daniel Vetter

Well I mostly run --strict since that checks for the alignment of continuation lines for function parameters. Imo not following the i915 coding style for that looks a bit too jarring ;-)

Still checkpatch is just a tool and occasionally seriously lacks taste, so please use --strict wisely instead of strictly.

Perhaps we should use the --subjective option instead. Not that it would actually change anything, it's exactly the same as --strict. We'd still require the same things for i915, but we'd be openly subjective about it. ;)

BR,
Jani.

Signature · 5 lines
Jani Nikula, Intel Open Source Technology Center
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx