Add enlightenments for vGPU
Subsystems (3)
v3 cover Add enlightenments for vGPU
Intel GVT-g (previously known as XenGT), is a complete GPU virtualization solution with mediated pass-through for 4th generation Intel Core processors - Haswell platform. This technology presents a virtual full-fledged GPU to each Virtual Machine (VM). VMs can directly access performance-critical resources, without intervention from the hypervisor in most cases, while privileged operations from VMs are trap-and-emulated at minimal cost. For detail, please refer to
https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-release
This patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, will be sent out in another patchset.
The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory w/o hypervisor's intervention, e.g. filling textures or queuing commands. However w/ the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses w/o complexity of translation.
Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (w/ some faulting mechanism).
The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource XenGT also partitions them among VMs.
Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.
Yu Zhang (8):
drm/i915: Introduce a PV INFO page structure for Intel GVT-g. drm/i915: Adds graphic address space ballooning logic drm/i915: Partition the fence registers for vGPU in i915 driver drm/i915: Disable framebuffer compression for i915 driver in VM drm/i915: Add the display switch logic for vGPU in i915 driver drm/i915: Disable power management for i915 driver in VM drm/i915: Create vGPU specific write MMIO to reduce traps drm/i915: Support alias ppgtt in VM if ppgtt is enabled
Documentation/DocBook/drm.tmpl | 5 + drivers/gpu/drm/i915/Makefile | 3 + drivers/gpu/drm/i915/i915_dma.c | 11 ++ drivers/gpu/drm/i915/i915_drv.h | 11 ++ drivers/gpu/drm/i915/i915_gem.c | 5 + drivers/gpu/drm/i915/i915_gem_gtt.c | 33 ++++- drivers/gpu/drm/i915/i915_vgpu.c | 234 ++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 94 +++++++++++++++ drivers/gpu/drm/i915/intel_pm.c | 8 ++ drivers/gpu/drm/i915/intel_uncore.c | 19 +++ 10 files changed, 420 insertions(+), 3 deletions(-) create mode 100644 drivers/gpu/drm/i915/i915_vgpu.c create mode 100644 drivers/gpu/drm/i915/i915_vgpu.h
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v3 1/8 drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through. This page contains the shared information between i915 driver and the host emulator. For now, this structure utilizes an area of 4K bypes on HSW GPU's unused MMIO space to support existing production. Future hardware will have the reserved window architecturally defined, and layout of the page will be added in future BSpec.
The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. Thereafter a flag, vgpu.active is set, and intel_vgpu_active() is used by checking this flag to conclude if GPU is virtualized with Intel GVT-g. By now, intel_vgpu_active() will return true, only when the driver is running as a guest in the Intel GVT-g enhanced environment on HSW platform.
v2:
take Chris' comments:
- call the i915_check_vgpu() in intel_uncore_init()
- sanitize i915_check_vgpu() by adding BUILD_BUG_ON() and debug info
take Daniel's comments:
- put the definition of PV INFO into a new header - i915_vgt_if.h
other changes:
- access mmio regs by readq/readw in i915_check_vgpu()
v3:
take Daniel's comments:
- move the i915/vgt interfaces into a new i915_vgpu.c
- update makefile
- add kerneldoc to functions which are non-static
- add a DOC: section describing some of the high-level design
- update drm docbook
other changes:
- rename i915_vgt_if.h to i915_vgpu.h
--- Documentation/DocBook/drm.tmpl | 5 +++ drivers/gpu/drm/i915/Makefile | 3 ++ drivers/gpu/drm/i915/i915_drv.h | 11 +++++ drivers/gpu/drm/i915/i915_vgpu.c | 85 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 85 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/intel_uncore.c | 3 ++ 6 files changed, 192 insertions(+) create mode 100644 drivers/gpu/drm/i915/i915_vgpu.c create mode 100644 drivers/gpu/drm/i915/i915_vgpu.h
Documentation/DocBook/drm.tmpl +5 -0
@@ -3832,6 +3832,11 @@ int num_ioctls;</synopsis> !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_disable_interrupts !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_enable_interrupts </sect2> + <sect2> + <title>Intel GVT-g Guest Support(vGPU)</title> +!Pdrivers/gpu/drm/i915/i915_vgpu.c Intel GVT-g guest support +!Idrivers/gpu/drm/i915/i915_vgpu.c + </sect2> </sect1> <sect1> <title>Display Hardware Handling</title>
drivers/gpu/drm/i915/Makefile +3 -0
@@ -79,6 +79,9 @@ i915-y += dvo_ch7017.o \ intel_sdvo.o \ intel_tv.o +# virtual gpu code +i915-y += i915_vgpu.o + # legacy horrors i915-y += i915_dma.o \ i915_ums.o
drivers/gpu/drm/i915/i915_drv.h +11 -0
@@ -1470,6 +1470,10 @@ struct i915_workarounds {u32 count; }; +struct i915_virtual_gpu { +bool active; +}; +struct drm_i915_private {struct drm_device *dev;struct kmem_cache *slab;
@@ -1482,6 +1486,8 @@ struct drm_i915_private {struct intel_uncore uncore; +struct i915_virtual_gpu vgpu; +struct intel_gmbus gmbus[GMBUS_NUM_PORTS ];
@@ -2311,6 +2317,11 @@ extern void intel_uncore_check_errors(struct drm_device *dev);extern void intel_uncore_fini (struct drm_device *dev);extern void intel_uncore_forcewake_reset (struct drm_device *dev,bool restore); +static inline bool intel_vgpu_active (struct drm_device *dev) +{ +return to_i915 (dev)->vgpu.active; +} +void i915_enable_pipestat (struct drm_i915_private *dev_priv,enum pipe pipe,u32 status_mask);
drivers/gpu/drm/i915/i915_vgpu.c new file +85 -0
85 lines not on this page · raw message
drivers/gpu/drm/i915/i915_vgpu.h new file +85 -0
85 lines not on this page · raw message
drivers/gpu/drm/i915/intel_uncore.c +3 -0
@@ -23,6 +23,7 @@#include "i915_drv.h" #include "intel_drv.h" +#include "i915_vgpu.h" #define FORCEWAKE_ACK_TIMEOUT_MS 2
@@ -850,6 +851,8 @@ void intel_uncore_init(struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private; +i915_check_vgpu (dev); +setup_timer (&dev_priv->uncore.force_wake_timer, gen6_force_wake_timer, (unsigned long )dev_priv);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
Quoted 5 lines · Yu Zhang
Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through. This page contains the shared information between i915 driver and the host emulator. For now, this structure utilizes an area of 4K bypes on HSW GPU's unused MMIO space to support
bytes
Quoted 1 line
existing production. Future hardware will have the reserved window
It is slightly unclear to me what "existing production" means?
Quoted 148 lines
architecturally defined, and layout of the page will be added in future BSpec.
The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. Thereafter a flag, vgpu.active is set, and
intel_vgpu_active()is used by checking this flag to conclude if GPU is virtualized with Intel GVT-g. By now,intel_vgpu_active()will return true, only when the driver is running as a guest in the Intel GVT-g enhanced environment on HSW platform.v2:
take Chris' comments:
- call thei915_check_vgpu()inintel_uncore_init()
- sanitizei915_check_vgpu()by addingBUILD_BUG_ON()and debug info
take Daniel's comments:
- put the definition of PV INFO into a new header -i915_vgt_if.h
other changes:
- access mmio regs by readq/readw ini915_check_vgpu()v3:
take Daniel's comments:
- move the i915/vgt interfaces into a newi915_vgpu.c
- update makefile
- add kerneldoc to functions which are non-static
- add a DOC: section describing some of the high-level design
- update drm docbook
other changes:
- renamei915_vgt_if.htoi915_vgpu.hSigned-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Eddie Dong <[email protected]>--- Documentation/DocBook/drm.tmpl | 5 +++ drivers/gpu/drm/i915/Makefile | 3 ++ drivers/gpu/drm/i915/i915_drv.h | 11 +++++ drivers/gpu/drm/i915/i915_vgpu.c | 85 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 85 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/intel_uncore.c | 3 ++ 6 files changed, 192 insertions(+)create mode 100644
drivers/gpu/drm/i915/i915_vgpu.ccreate mode 100644drivers/gpu/drm/i915/i915_vgpu.hDocumentation/DocBook/drm.tmpl +5
-0@@ -3832,6 +3832,11 @@ int num_ioctls;</synopsis> !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_disable_interrupts !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_enable_interrupts </sect2> + <sect2> + <title>Intel GVT-g Guest Support(vGPU)</title> +!Pdrivers/gpu/drm/i915/i915_vgpu.c Intel GVT-g guest support +!Idrivers/gpu/drm/i915/i915_vgpu.c + </sect2> </sect1> <sect1> <title>Display Hardware Handling</title>drivers/gpu/drm/i915/Makefile +3
-0@@ -79,6 +79,9 @@ i915-y += dvo_ch7017.o \ intel_sdvo.o \ intel_tv.o +# virtual gpu code +i915-y += i915_vgpu.o + # legacy horrors i915-y += i915_dma.o \ i915_ums.odrivers/gpu/drm/i915/i915_drv.h +11
-0@@ -1470,6 +1470,10 @@ struct i915_workarounds {u32 count; }; +struct i915_virtual_gpu { +bool active; +}; +struct drm_i915_private {struct drm_device *dev;struct kmem_cache *slab;@@ -1482,6 +1486,8 @@ struct drm_i915_private {struct intel_uncore uncore; +struct i915_virtual_gpu vgpu; +struct intel_gmbus gmbus[GMBUS_NUM_PORTS ];@@ -2311,6 +2317,11 @@ extern void intel_uncore_check_errors(struct drm_device *dev);extern void intel_uncore_fini (struct drm_device *dev);extern void intel_uncore_forcewake_reset (struct drm_device *dev,bool restore); +static inline bool intel_vgpu_active (struct drm_device *dev) +{ +return to_i915 (dev)->vgpu.active; +} +void i915_enable_pipestat (struct drm_i915_private *dev_priv,enum pipe pipe,u32 status_mask);drivers/gpu/drm/i915/i915_vgpu.c new file +32
-0@@ -0,0 +1,85 @@ +/* +* Copyright(c) 2011-2014 Intel Corporation. All rights reserved. +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice (including the next +* paragraph) shall be included in all copies or substantial portions of the +* Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +* SOFTWARE. +*/ + +#include "intel_drv.h" +#include "i915_vgpu.h" + + +/** +* DOC: Intel GVT-g guest support +* +* Intel GVT-g is a graphics virtualization technology which shares the +* GPU among multiple virtual machines on a time-sharing basis. Each
drivers/gpu/drm/i915/i915_vgpu.c line 42
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +10
-0+* virtual machine is presented a virtual GPU (vGPU), which has equivalent +* features as the underlying physical GPU (pGPU), so i915 driver can run +* seamlessly in a virtual machine. This file provides vGPU specific +* optimizations when running in a virtual machine, to reduce the complexity +* of vGPU emulation and to improve the overall performance. +* +* A primary function introduced here is so-called "address space ballooning" +* technique. Intel GVT-g partitions global graphics memory among multiple VMs, +* so each VM can directly access a portion of the memory w/o hypervisor's +* intervention, e.g. filling textures or queuing commands. However w/ the
Personal preference, but It may be nicer to avoid "w/o" and similar shorthand in documentation.
Quoted 29 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +29
-0+* partitioning an unmodified i915 driver would assume a smaller graphics +* memory starting from address ZERO, then requires vGPU emulation module to +* translate the graphics address between 'guest view' and 'host view', for +* all registers and command opcodes which contain a graphics memory address. +* To reduce the complexity, Intel GVT-g introduces "address space ballooning", +* by telling the exact partitioning knowledge to each guest i915 driver, which +* then reserves and prevents non-allocated portions from allocation. Thus vGPU +* emulation module module only needs to scan and validate graphics addresses +* w/o complexity of address translation. +* +*/ + +/** +* i915_check_vgpu - detect virtual GPU +* @dev: drm device * +* +* This function is called at the initialization stage, to detect whether +* running on a vGPU. +*/ +void i915_check_vgpu (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv =to_i915 (dev); +uint64_t magic; +uint32_t version; + +BUILD_BUG_ON (sizeof (struct vgt_if ) !=VGT_PVINFO_SIZE ); + +if (!IS_HASWELL (dev)) +return ;
drivers/gpu/drm/i915/i915_vgpu.c line 81
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +10
-0+ + magic =readq (dev_priv->regs +vgtif_reg (magic)); +if (magic !=VGT_MAGIC ) +return ; + + version =INTEL_VGT_IF_VERSION_ENCODE ( +readw (dev_priv->regs +vgtif_reg (version_major)), +readw (dev_priv->regs +vgtif_reg (version_minor))); +if (version !=INTEL_VGT_IF_VERSION ) +return ;
Would it be useful to something on version mismatch?
Quoted 47 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +3
-0+ dev_priv->vgpu.active = true; +DRM_INFO ("Virtual GPU for Intel GVT-g detected.\n" ); +}drivers/gpu/drm/i915/i915_vgpu.h new file +38
-0@@ -0,0 +1,85 @@ +/* +* Copyright(c) 2011-2014 Intel Corporation. All rights reserved. +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice (including the next +* paragraph) shall be included in all copies or substantial portions of the +* Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +* SOFTWARE. +*/ + +#ifndef _I915_VGPU_H_ +#define _I915_VGPU_H_ + +/* The MMIO offset of the shared info between guest and host emulator */ +#define VGT_PVINFO_PAGE 0x78000 +#define VGT_PVINFO_SIZE 0x1000 + +/* +* The following structure pages are defined in GEN MMIO space +* for virtualization. (One page for now) +*/ +#define VGT_MAGIC 0x4776544776544776/* 'vGTvGTvG' */ +#define VGT_VERSION_MAJOR 1 +#define VGT_VERSION_MINOR 0 +
drivers/gpu/drm/i915/i915_vgpu.h line 48
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +10
-0+#define INTEL_VGT_IF_VERSION_ENCODE (major, minor) ((major) << 16 | (minor)) +#define INTEL_VGT_IF_VERSION \ + INTEL_VGT_IF_VERSION_ENCODE(VGT_VERSION_MAJOR, VGT_VERSION_MINOR) + +struct vgt_if { +uint64_t magic;/* VGT_MAGIC */ +uint16_t version_major; +uint16_t version_minor; +uint32_t vgt_id;/* ID of vGT instance */ +uint32_t rsv1[12 ];/* pad to offset 0x40 */
In general I thought stdint types should be avoided in the kernel in favour of u32 and company. But I see i915 is full of them already so maybe my information is outdated.
Quoted 11 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +11
-0+/* +* Data structure to describe the balooning info of resources. +* Each VM can only have one portion of continuous area for now. +* (May support scattered resource in future) +* (starting from offset 0x40) +*/ + struct { + /* Aperture register balooning */ + struct { + uint32_t my_base; + uint32_t my_size;
drivers/gpu/drm/i915/i915_vgpu.h line 69
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +10
-0+ } low_gmadr;/* aperture */ +/* GMADR register balooning */ +struct { +uint32_t my_base; +uint32_t my_size; + } high_gmadr;/* non aperture */ +/* allowed fence registers */ +uint32_t fence_num; +uint32_t rsv2[3 ]; + } avail_rs;/* available/assigned resource */
I didn't figure out yet why the concept is to pass in two ranges to mark as reserved, instead of one usable/allowed range and then implicitly all the rest as reserved?
drivers/gpu/drm/i915/i915_vgpu.h line 78
Quoted 9 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +9
-0+uint32_t rsv3[0x200 -24 ];/* pad to half page */ +/* +* The bottom half page is for response from Gfx driver to hypervisor. +* Set to reserved fields temporarily by now. +*/ +uint32_t rsv4; +uint32_t display_ready;/* ready for display owner switch */ +uint32_t rsv5[0x200 -2 ];/* pad to one page */ +} __packed;
You probably don't need __packed since you have BUILD_BUG_ON on size mismatch and pad to full size with reserved fields. I don't think it matters though.
Quoted 20 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +6
-0+#define vgtif_reg (x) \ + (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x) + +extern void i915_check_vgpu (struct drm_device *dev); + +#endif /* _I915_VGPU_H_ */ drivers/gpu/drm/i915/intel_uncore.c +1
-0@@ -23,6 +23,7 @@#include "i915_drv.h" #include "intel_drv.h" +#include "i915_vgpu.h" #define FORCEWAKE_ACK_TIMEOUT_MS 2
drivers/gpu/drm/i915/intel_uncore.c void intel_uncore_init(struct drm_device *dev) · line 858
Quoted 8 lines
drivers/gpu/drm/i915/intel_uncore.c +2
-0@@ -850,6 +851,8 @@ void intel_uncore_init(struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private; +i915_check_vgpu (dev); +setup_timer (&dev_priv->uncore.force_wake_timer, gen6_force_wake_timer, (unsigned long )dev_priv);
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
Quoted 10 lines · Tvrtko Ursulin
drivers/gpu/drm/i915/i915_vgpu.h line 48
Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +6
-0+struct vgt_if { +uint64_t magic;/* VGT_MAGIC */ +uint16_t version_major; +uint16_t version_minor; +uint32_t vgt_id;/* ID of vGT instance */ +uint32_t rsv1[12 ];/* pad to offset 0x40 */ In general I thought stdint types should be avoided in the kernel in favour of u32 and company. But I see i915 is full of them already so maybe my information is outdated.
That was back when c99 types weren't a common thing, nowadays I think we should prefer _t types for more uniformity with userspace. Also it makes the __u32 and similar types which are required for ioctl structs stand out much more. And yes we're totally inconsistent about it.
Just an aside.
-Daniel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
In reply to Tvrtko UrsulinThank you very much for your thorough review, Tvrtko. :)
Quoted 9 lines · Tvrtko Ursulin
Quoted 5 lines · Yu Zhang
Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through. This page contains the shared information between i915 driver and the host emulator. For now, this structure utilizes an area of 4K bypes on HSW GPU's unused MMIO space to support
bytes
Got it. Thanks.
Quoted 4 lines
Quoted 2 lines
existing production. Future hardware will have the reserved window
It is slightly unclear to me what "existing production" means?
Our current gpu virtualization implementation for HSW. It's compared with future implematations for BDW etc.
Quoted 181 lines
Quoted 181 lines
architecturally defined, and layout of the page will be added in future BSpec.
The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. Thereafter a flag, vgpu.active is set, and
intel_vgpu_active()is used by checking this flag to conclude if GPU is virtualized with Intel GVT-g. By now,intel_vgpu_active()will return true, only when the driver is running as a guest in the Intel GVT-g enhanced environment on HSW platform.v2:
take Chris' comments:
- call thei915_check_vgpu()inintel_uncore_init()
- sanitizei915_check_vgpu()by addingBUILD_BUG_ON()and debug info
take Daniel's comments:
- put the definition of PV INFO into a new header -i915_vgt_if.h
other changes:
- access mmio regs by readq/readw ini915_check_vgpu()v3:
take Daniel's comments:
- move the i915/vgt interfaces into a newi915_vgpu.c
- update makefile
- add kerneldoc to functions which are non-static
- add a DOC: section describing some of the high-level design
- update drm docbook
other changes:
- renamei915_vgt_if.htoi915_vgpu.hSigned-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Eddie Dong <[email protected]>--- Documentation/DocBook/drm.tmpl | 5 +++ drivers/gpu/drm/i915/Makefile | 3 ++ drivers/gpu/drm/i915/i915_drv.h | 11 +++++ drivers/gpu/drm/i915/i915_vgpu.c | 85 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 85 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/intel_uncore.c | 3 ++ 6 files changed, 192 insertions(+)create mode 100644
drivers/gpu/drm/i915/i915_vgpu.ccreate mode 100644drivers/gpu/drm/i915/i915_vgpu.hDocumentation/DocBook/drm.tmpl +5
-0@@ -3832,6 +3832,11 @@ int num_ioctls;</synopsis> !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_disable_interrupts !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_enable_interrupts </sect2> + <sect2> + <title>Intel GVT-g Guest Support(vGPU)</title> +!Pdrivers/gpu/drm/i915/i915_vgpu.c Intel GVT-g guest support +!Idrivers/gpu/drm/i915/i915_vgpu.c + </sect2> </sect1> <sect1> <title>Display Hardware Handling</title>drivers/gpu/drm/i915/Makefile +3
-0@@ -79,6 +79,9 @@ i915-y += dvo_ch7017.o \ intel_sdvo.o \ intel_tv.o +# virtual gpu code +i915-y += i915_vgpu.o + # legacy horrors i915-y += i915_dma.o \ i915_ums.odrivers/gpu/drm/i915/i915_drv.h +11
-0@@ -1470,6 +1470,10 @@ struct i915_workarounds {u32 count; }; +struct i915_virtual_gpu { +bool active; +}; +struct drm_i915_private {struct drm_device *dev;struct kmem_cache *slab;@@ -1482,6 +1486,8 @@ struct drm_i915_private {struct intel_uncore uncore; +struct i915_virtual_gpu vgpu; +struct intel_gmbus gmbus[GMBUS_NUM_PORTS ];@@ -2311,6 +2317,11 @@ extern void intel_uncore_check_errors(struct drm_device *dev);extern void intel_uncore_fini (struct drm_device *dev);extern void intel_uncore_forcewake_reset (struct drm_device *dev,bool restore); +static inline bool intel_vgpu_active (struct drm_device *dev) +{ +return to_i915 (dev)->vgpu.active; +} +void i915_enable_pipestat(struct
drm_i915_private*dev_priv, enum pipe pipe,u32 status_mask);drivers/gpu/drm/i915/i915_vgpu.c new file +4
-0@@ -0,0 +1,85 @@ +/* +* Copyright(c) 2011-2014 Intel Corporation. All rights reserved. +* +* Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */drivers/gpu/drm/i915/i915_vgpu.c new file +11
-0+ +#include "intel_drv.h" +#include "i915_vgpu.h" + + +/** +* DOC: Intel GVT-g guest support +* +* Intel GVT-g is a graphics virtualization technology which shares the +* GPU among multiple virtual machines on a time-sharing basis. Each +* virtual machine is presented a virtual GPU (vGPU), which has equivalent
+ * features as the underlying physical GPU (pGPU), so i915 driver can run
+ * seamlessly in a virtual machine. This file provides vGPU specific
+ * optimizations when running in a virtual machine, to reduce the complexity
+ * of vGPU emulation and to improve the overall performance.
+ *
+ * A primary function introduced here is so-called "address space ballooning"
+ * technique. Intel GVT-g partitions global graphics memory among
Quoted 8 lines
Quoted 5 lines
multiple VMs,
+ * so each VM can directly access a portion of the memory w/o hypervisor's
+ * intervention, e.g. filling textures or queuing commands. However w/ thePersonal preference, but It may be nicer to avoid "w/o" and similar shorthand in documentation.
OK. Will fix it. :)
Quoted 43 lines
Quoted 43 lines
+ * partitioning an unmodified i915 driver would assume a smaller graphics
+ * memory starting from address ZERO, then requires vGPU emulation module to
+ * translate the graphics address between 'guest view' and 'host view', for
+ * all registers and command opcodes which contain a graphics memory address.
+ * To reduce the complexity, Intel GVT-g introduces "address space ballooning",
+ * by telling the exact partitioning knowledge to each guest i915 driver, which
+ * then reserves and prevents non-allocated portions from allocation. Thus vGPU
+ * emulation module module only needs to scan and validate graphics addresses
+ * w/o complexity of address translation.
+ *
+ */drivers/gpu/drm/i915/i915_vgpu.c new file +6
-0+ +/** +* i915_check_vgpu - detect virtual GPU +* @dev: drm device * +* +* This function is called at the initialization stage, to detect whether
+ * running on a vGPU.
+ */drivers/gpu/drm/i915/i915_vgpu.c new file +14
-0+void i915_check_vgpu (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv =to_i915 (dev); +uint64_t magic; +uint32_t version; + +BUILD_BUG_ON (sizeof (struct vgt_if ) !=VGT_PVINFO_SIZE ); + +if (!IS_HASWELL (dev)) +return ; + + magic =readq (dev_priv->regs +vgtif_reg (magic)); +if (magic !=VGT_MAGIC ) +return ;
Quoted 8 lines
drivers/gpu/drm/i915/i915_vgpu.c line 81
Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +6
-0+ + version =INTEL_VGT_IF_VERSION_ENCODE ( +readw (dev_priv->regs +vgtif_reg (version_major)), +readw (dev_priv->regs +vgtif_reg (version_minor))); +if (version !=INTEL_VGT_IF_VERSION ) +return ;Would it be useful to something on version mismatch?
Well. Thanks for your notification. :)
In fact, this version judgement is just for future usages. By now, we have only one vgt interface version, 0x10000. But you are right, unlike the judgement for 'magic', which specifies the Intel GVT-g feature, and can be returned directly when false, failure for the 'version' may should trigger something like unload the driver. However, if so, this means we may need to:
1> return something in i915_check_vgpu();
2> change the prototype of i915_check_vgpu() and its caller intel_uncore_init();
3> check the return value of intel_uncore_init() in i915_driver_load()
Is this approach acceptable?
Quoted 68 lines
Quoted 68 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +3
-0+ dev_priv->vgpu.active = true; +DRM_INFO ("Virtual GPU for Intel GVT-g detected.\n" ); +}drivers/gpu/drm/i915/i915_vgpu.h new file +4
-0@@ -0,0 +1,85 @@ +/* +* Copyright(c) 2011-2014 Intel Corporation. All rights reserved. +* +* Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */drivers/gpu/drm/i915/i915_vgpu.h new file +17
-0+ +#ifndef _I915_VGPU_H_ +#define _I915_VGPU_H_ + +/* The MMIO offset of the shared info between guest and host emulator */ +#define VGT_PVINFO_PAGE 0x78000 +#define VGT_PVINFO_SIZE 0x1000 + +/* +* The following structure pages are defined in GEN MMIO space +* for virtualization. (One page for now) +*/ +#define VGT_MAGIC 0x4776544776544776/* 'vGTvGTvG' */ +#define VGT_VERSION_MAJOR 1 +#define VGT_VERSION_MINOR 0 + +#define INTEL_VGT_IF_VERSION_ENCODE (major, minor) ((major) << 16 |(minor))
drivers/gpu/drm/i915/i915_vgpu.h new file +5
-0+#define INTEL_VGT_IF_VERSION \ + INTEL_VGT_IF_VERSION_ENCODE(VGT_VERSION_MAJOR, VGT_VERSION_MINOR) + +struct vgt_if { +uint64_t magic;/* VGT_MAGIC */
Quoted 9 lines
drivers/gpu/drm/i915/i915_vgpu.h line 48
Quoted 4 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +4
-0+uint16_t version_major; +uint16_t version_minor; +uint32_t vgt_id;/* ID of vGT instance */ +uint32_t rsv1[12 ];/* pad to offset 0x40 */ In general I thought stdint types should be avoided in the kernel in favour of u32 and company. But I see i915 is full of them already so maybe my information is outdated.
Let's keep the _t types, like explanations in Daniel's mail. Is this OK?
Quoted 22 lines
Quoted 11 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +11
-0+/* +* Data structure to describe the balooning info of resources. +* Each VM can only have one portion of continuous area for now. +* (May support scattered resource in future) +* (starting from offset 0x40) +*/ +struct { +/* Aperture register balooning */ +struct { +uint32_t my_base; +uint32_t my_size;drivers/gpu/drm/i915/i915_vgpu.h line 69
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +10
-0+ } low_gmadr;/* aperture */ +/* GMADR register balooning */ +struct { +uint32_t my_base; +uint32_t my_size; + } high_gmadr;/* non aperture */ +/* allowed fence registers */ +uint32_t fence_num; +uint32_t rsv2[3 ]; + } avail_rs;/* available/assigned resource */
Quoted 4 lines
I didn't figure out yet why the concept is to pass in two ranges to mark as reserved, instead of one usable/allowed range and then implicitly all the rest as reserved?
Like you said in 2/3 comments, this 2 part is for mappable and non-mapplable respectively. :)
Quoted 11 lines
drivers/gpu/drm/i915/i915_vgpu.h line 78
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +3
-0+uint32_t rsv3[0x200 -24 ];/* pad to half page */ +/* +* The bottom half page is for response from Gfx driver to hypervisor.
drivers/gpu/drm/i915/i915_vgpu.h new file +6
-0+* Set to reserved fields temporarily by now. +*/ +uint32_t rsv4; +uint32_t display_ready;/* ready for display owner switch */ +uint32_t rsv5[0x200 -2 ];/* pad to one page */ +} __packed;
Quoted 4 lines
You probably don't need
__packedsince you haveBUILD_BUG_ONon size mismatch and pad to full size with reserved fields. I don't think it matters though.
Well, IIRC, this __packed is added to follow Chris's comments. How about we keep it? Or will it hurt any place? :)
Quoted 32 lines
Quoted 21 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +6
-0+#define vgtif_reg (x) \ + (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x) + +extern void i915_check_vgpu (struct drm_device *dev); + +#endif /* _I915_VGPU_H_ */ drivers/gpu/drm/i915/intel_uncore.c +1
-0@@ -23,6 +23,7 @@#include "i915_drv.h" #include "intel_drv.h" +#include "i915_vgpu.h" #define FORCEWAKE_ACK_TIMEOUT_MS 2drivers/gpu/drm/i915/intel_uncore.c void intel_uncore_init(struct drm_device *dev) · line 858
Quoted 8 lines
drivers/gpu/drm/i915/intel_uncore.c +2
-0@@ -850,6 +851,8 @@ void intel_uncore_init(struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private; +i915_check_vgpu (dev); +setup_timer (&dev_priv->uncore.force_wake_timer, gen6_force_wake_timer, (unsigned long )dev_priv);Regards,
Quoted 1 line
Tvrtko
B.R.
Yu
Quoted 2 lines
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
Quoted 11 lines · Zhang
Thank you very much for your thorough review, Tvrtko. :)
Quoted 8 lines · Tvrtko Ursulin
Quoted 5 lines · Yu Zhang
Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through. This page contains the shared information between i915 driver and the host emulator. For now, this structure utilizes an area of 4K bypes on HSW GPU's unused MMIO space to support
Quoted 8 lines
Quoted 1 line
bytes
Got it. Thanks.
Quoted 4 lines
Quoted 2 lines
existing production. Future hardware will have the reserved window
It is slightly unclear to me what "existing production" means?
Our current gpu virtualization implementation for HSW. It's compared with future implematations for BDW etc.
So maybe instead of "existing production" say "currently supported hardware" or similar?
Quoted 242 lines
Quoted 182 lines
Quoted 182 lines
architecturally defined, and layout of the page will be added in future BSpec.
The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. Thereafter a flag, vgpu.active is set, and
intel_vgpu_active()is used by checking this flag to conclude if GPU is virtualized with Intel GVT-g. By now,intel_vgpu_active()will return true, only when the driver is running as a guest in the Intel GVT-g enhanced environment on HSW platform.v2:
take Chris' comments:
- call thei915_check_vgpu()inintel_uncore_init()
- sanitizei915_check_vgpu()by addingBUILD_BUG_ON()and debug info
take Daniel's comments:
- put the definition of PV INFO into a new header -i915_vgt_if.h
other changes:
- access mmio regs by readq/readw ini915_check_vgpu()v3:
take Daniel's comments:
- move the i915/vgt interfaces into a newi915_vgpu.c
- update makefile
- add kerneldoc to functions which are non-static
- add a DOC: section describing some of the high-level design
- update drm docbook
other changes:
- renamei915_vgt_if.htoi915_vgpu.hSigned-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Eddie Dong <[email protected]>--- Documentation/DocBook/drm.tmpl | 5 +++ drivers/gpu/drm/i915/Makefile | 3 ++ drivers/gpu/drm/i915/i915_drv.h | 11 +++++ drivers/gpu/drm/i915/i915_vgpu.c | 85 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 85 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/intel_uncore.c | 3 ++ 6 files changed, 192 insertions(+)create mode 100644
drivers/gpu/drm/i915/i915_vgpu.ccreate mode 100644drivers/gpu/drm/i915/i915_vgpu.hDocumentation/DocBook/drm.tmpl +5
-0@@ -3832,6 +3832,11 @@ int num_ioctls;</synopsis> !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_disable_interrupts !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_enable_interrupts </sect2> + <sect2> + <title>Intel GVT-g Guest Support(vGPU)</title> +!Pdrivers/gpu/drm/i915/i915_vgpu.c Intel GVT-g guest support +!Idrivers/gpu/drm/i915/i915_vgpu.c + </sect2> </sect1> <sect1> <title>Display Hardware Handling</title>drivers/gpu/drm/i915/Makefile +3
-0@@ -79,6 +79,9 @@ i915-y += dvo_ch7017.o \ intel_sdvo.o \ intel_tv.o +# virtual gpu code +i915-y += i915_vgpu.o + # legacy horrors i915-y += i915_dma.o \ i915_ums.odrivers/gpu/drm/i915/i915_drv.h +11
-0@@ -1470,6 +1470,10 @@ struct i915_workarounds {u32 count; }; +struct i915_virtual_gpu { +bool active; +}; +struct drm_i915_private {struct drm_device *dev;struct kmem_cache *slab;@@ -1482,6 +1486,8 @@ struct drm_i915_private {struct intel_uncore uncore; +struct i915_virtual_gpu vgpu; +struct intel_gmbus gmbus[GMBUS_NUM_PORTS ];@@ -2311,6 +2317,11 @@ extern void intel_uncore_check_errors(struct drm_device *dev);extern void intel_uncore_fini (struct drm_device *dev);extern void intel_uncore_forcewake_reset (struct drm_device *dev,bool restore); +static inline bool intel_vgpu_active (struct drm_device *dev) +{ +return to_i915 (dev)->vgpu.active; +} +void i915_enable_pipestat(struct
drm_i915_private*dev_priv, enum pipe
pipe,u32 status_mask);drivers/gpu/drm/i915/i915_vgpu.c new file +4
-0@@ -0,0 +1,85 @@ +/* +* Copyright(c) 2011-2014 Intel Corporation. All rights reserved. +* +* Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */drivers/gpu/drm/i915/i915_vgpu.c new file +11
-0+ +#include "intel_drv.h" +#include "i915_vgpu.h" + + +/** +* DOC: Intel GVT-g guest support +* +* Intel GVT-g is a graphics virtualization technology which shares the +* GPU among multiple virtual machines on a time-sharing basis. Each +* virtual machine is presented a virtual GPU (vGPU), which has equivalent
+ * features as the underlying physical GPU (pGPU), so i915 driver can run
+ * seamlessly in a virtual machine. This file provides vGPU specific
+ * optimizations when running in a virtual machine, to reduce the complexity
+ * of vGPU emulation and to improve the overall performance.
+ *
+ * A primary function introduced here is so-called "address space ballooning"
+ * technique. Intel GVT-g partitions global graphics memory amongQuoted 8 lines
Quoted 5 lines
multiple VMs,
+ * so each VM can directly access a portion of the memory w/o hypervisor's
+ * intervention, e.g. filling textures or queuing commands. However w/ thePersonal preference, but It may be nicer to avoid "w/o" and similar shorthand in documentation.
OK. Will fix it. :)
Quoted 43 lines
Quoted 43 lines
+ * partitioning an unmodified i915 driver would assume a smaller graphics
+ * memory starting from address ZERO, then requires vGPU emulation module to
+ * translate the graphics address between 'guest view' and 'host view', for
+ * all registers and command opcodes which contain a graphics memory address.
+ * To reduce the complexity, Intel GVT-g introduces "address space ballooning",
+ * by telling the exact partitioning knowledge to each guest i915 driver, which
+ * then reserves and prevents non-allocated portions from allocation. Thus vGPU
+ * emulation module module only needs to scan and validate graphics addresses
+ * w/o complexity of address translation.
+ *
+ */drivers/gpu/drm/i915/i915_vgpu.c new file +6
-0+ +/** +* i915_check_vgpu - detect virtual GPU +* @dev: drm device * +* +* This function is called at the initialization stage, to detect whether
+ * running on a vGPU.
+ */drivers/gpu/drm/i915/i915_vgpu.c new file +14
-0+void i915_check_vgpu (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv =to_i915 (dev); +uint64_t magic; +uint32_t version; + +BUILD_BUG_ON (sizeof (struct vgt_if ) !=VGT_PVINFO_SIZE ); + +if (!IS_HASWELL (dev)) +return ; + + magic =readq (dev_priv->regs +vgtif_reg (magic)); +if (magic !=VGT_MAGIC ) +return ;Quoted 8 lines
drivers/gpu/drm/i915/i915_vgpu.c line 81
Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +6
-0+ + version =INTEL_VGT_IF_VERSION_ENCODE ( +readw (dev_priv->regs +vgtif_reg (version_major)), +readw (dev_priv->regs +vgtif_reg (version_minor))); +if (version !=INTEL_VGT_IF_VERSION ) +return ;Would it be useful to something on version mismatch?
Quoted 12 lines
Well. Thanks for your notification. :)
In fact, this version judgement is just for future usages. By now, we have only one vgt interface version,0x10000. But you are right, unlike the judgement for 'magic', which specifies the Intel GVT-g feature, and can be returned directly when false, failure for the 'version' may should trigger something like unload the driver. However, if so, this means we may need to:
1> return something ini915_check_vgpu();
2> change the prototype ofi915_check_vgpu()and its callerintel_uncore_init();
3> check the return value ofintel_uncore_init()ini915_driver_load()Is this approach acceptable?
I would leave this for the future version. I was simply suggesting logging something on version mismatch. Just because it is unexpected so there is a trace in the logs why feature is not available. It's up to you.
Quoted 79 lines
Quoted 70 lines
Quoted 70 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +3
-0+ dev_priv->vgpu.active = true; +DRM_INFO ("Virtual GPU for Intel GVT-g detected.\n" ); +}drivers/gpu/drm/i915/i915_vgpu.h new file +4
-0@@ -0,0 +1,85 @@ +/* +* Copyright(c) 2011-2014 Intel Corporation. All rights reserved. +* +* Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */drivers/gpu/drm/i915/i915_vgpu.h new file +5
-0+ +#ifndef _I915_VGPU_H_ +#define _I915_VGPU_H_ + +/* The MMIO offset of the shared info between guest and host emulator */
drivers/gpu/drm/i915/i915_vgpu.h new file +12
-0+#define VGT_PVINFO_PAGE 0x78000 +#define VGT_PVINFO_SIZE 0x1000 + +/* +* The following structure pages are defined in GEN MMIO space +* for virtualization. (One page for now) +*/ +#define VGT_MAGIC 0x4776544776544776/* 'vGTvGTvG' */ +#define VGT_VERSION_MAJOR 1 +#define VGT_VERSION_MINOR 0 + +#define INTEL_VGT_IF_VERSION_ENCODE (major, minor) ((major) << 16 |(minor))
drivers/gpu/drm/i915/i915_vgpu.h new file +5
-0+#define INTEL_VGT_IF_VERSION \ + INTEL_VGT_IF_VERSION_ENCODE(VGT_VERSION_MAJOR, VGT_VERSION_MINOR) + +struct vgt_if { +uint64_t magic;/* VGT_MAGIC */ Quoted 9 lines
drivers/gpu/drm/i915/i915_vgpu.h line 48
Quoted 4 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +4
-0+uint16_t version_major; +uint16_t version_minor; +uint32_t vgt_id;/* ID of vGT instance */ +uint32_t rsv1[12 ];/* pad to offset 0x40 */ In general I thought stdint types should be avoided in the kernel in favour of u32 and company. But I see i915 is full of them already so maybe my information is outdated.
Quoted 1 line
Let's keep the _t types, like explanations in Daniel's mail. Is this OK?
Yes.
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +21
-0+/* +* Data structure to describe the balooning info of resources. +* Each VM can only have one portion of continuous area for now. +* (May support scattered resource in future) +* (starting from offset 0x40) +*/ +struct { +/* Aperture register balooning */ +struct { +uint32_t my_base; +uint32_t my_size; + } low_gmadr;/* aperture */ +/* GMADR register balooning */ +struct { +uint32_t my_base; +uint32_t my_size; + } high_gmadr;/* non aperture */ +/* allowed fence registers */ +uint32_t fence_num; +uint32_t rsv2[3 ]; + } avail_rs;/* available/assigned resource */
Quoted 6 lines
Quoted 4 lines
I didn't figure out yet why the concept is to pass in two ranges to mark as reserved, instead of one usable/allowed range and then implicitly all the rest as reserved?
Like you said in 2/3 comments, this 2 part is for mappable and non-mapplable respectively. :)
Are high and low memory as terms known in the i915 space so far as synonyms for mappable and non-mappable?
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +3
-0+uint32_t rsv3[0x200 -24 ];/* pad to half page */ +/* +* The bottom half page is for response from Gfx driver to hypervisor.
drivers/gpu/drm/i915/i915_vgpu.h new file +6
-0+* Set to reserved fields temporarily by now. +*/ +uint32_t rsv4; +uint32_t display_ready;/* ready for display owner switch */ +uint32_t rsv5[0x200 -2 ];/* pad to one page */ +} __packed;
Quoted 6 lines
Quoted 4 lines
You probably don't need
__packedsince you haveBUILD_BUG_ONon size mismatch and pad to full size with reserved fields. I don't think it matters though.Well, IIRC, this
__packedis added to follow Chris's comments. How about we keep it? Or will it hurt any place? :)
The only purpose for packed here I can see is a double guard along the BUILD_BUG_ON. As I said I don't think it makes any difference so you can leave it in.
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
Quoted 16 lines · Tvrtko Ursulin
Quoted 14 lines · Zhang
Thank you very much for your thorough review, Tvrtko. :)
Quoted 9 lines · Tvrtko Ursulin
Quoted 5 lines · Yu Zhang
Introduce a PV INFO structure, to facilitate the Intel GVT-g technology, which is a GPU virtualization solution with mediated pass-through. This page contains the shared information between i915 driver and the host emulator. For now, this structure utilizes an area of 4K bypes on HSW GPU's unused MMIO space to support
bytes
Got it. Thanks.
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
existing production. Future hardware will have the reserved window
It is slightly unclear to me what "existing production" means?
Our current gpu virtualization implementation for HSW. It's compared with future implematations for BDW etc.
So maybe instead of "existing production" say "currently supported hardware" or similar?
How about we remove "to support existing production" in this sentence? The "currently supported hardware" seems a bit redundant. :)
Quoted 373 lines
Quoted 246 lines
Quoted 186 lines
Quoted 186 lines
architecturally defined, and layout of the page will be added in future BSpec.
The i915 driver load routine detects if it is running in a VM by reading the contents of this PV INFO page. Thereafter a flag, vgpu.active is set, and
intel_vgpu_active()is used by checking this flag to conclude if GPU is virtualized with Intel GVT-g. By now,intel_vgpu_active()will return true, only when the driver is running as a guest in the Intel GVT-g enhanced environment on HSW platform.v2:
take Chris' comments:
- call thei915_check_vgpu()inintel_uncore_init()
- sanitizei915_check_vgpu()by addingBUILD_BUG_ON()and debug
info
take Daniel's comments:
- put the definition of PV INFO into a new header -i915_vgt_if.h
other changes:
- access mmio regs by readq/readw ini915_check_vgpu()v3:
take Daniel's comments:
- move the i915/vgt interfaces into a newi915_vgpu.c
- update makefile
- add kerneldoc to functions which are non-static
- add a DOC: section describing some of the high-level design
- update drm docbook
other changes:
- renamei915_vgt_if.htoi915_vgpu.hSigned-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Eddie Dong <[email protected]>--- Documentation/DocBook/drm.tmpl | 5 +++ drivers/gpu/drm/i915/Makefile | 3 ++ drivers/gpu/drm/i915/i915_drv.h | 11 +++++ drivers/gpu/drm/i915/i915_vgpu.c | 85 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 85 +++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/intel_uncore.c | 3 ++ 6 files changed, 192 insertions(+)create mode 100644
drivers/gpu/drm/i915/i915_vgpu.ccreate mode 100644drivers/gpu/drm/i915/i915_vgpu.hDocumentation/DocBook/drm.tmpl +5
-0@@ -3832,6 +3832,11 @@ int num_ioctls;</synopsis> !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_disable_interrupts !Fdrivers/gpu/drm/i915/i915_irq.c intel_runtime_pm_enable_interrupts </sect2> + <sect2> + <title>Intel GVT-g Guest Support(vGPU)</title> +!Pdrivers/gpu/drm/i915/i915_vgpu.c Intel GVT-g guest support +!Idrivers/gpu/drm/i915/i915_vgpu.c + </sect2> </sect1> <sect1> <title>Display Hardware Handling</title>drivers/gpu/drm/i915/Makefile +3
-0@@ -79,6 +79,9 @@ i915-y += dvo_ch7017.o \ intel_sdvo.o \ intel_tv.o +# virtual gpu code +i915-y += i915_vgpu.o + # legacy horrors i915-y += i915_dma.o \ i915_ums.odrivers/gpu/drm/i915/i915_drv.h +11
-0@@ -1470,6 +1470,10 @@ struct i915_workarounds {u32 count; }; +struct i915_virtual_gpu { +bool active; +}; +struct drm_i915_private {struct drm_device *dev;struct kmem_cache *slab;@@ -1482,6 +1486,8 @@ struct drm_i915_private {struct intel_uncore uncore; +struct i915_virtual_gpu vgpu; +struct intel_gmbus gmbus[GMBUS_NUM_PORTS ];@@ -2311,6 +2317,11 @@ extern void intel_uncore_check_errors(struct drm_device *dev);extern void intel_uncore_fini (struct drm_device *dev);extern void intel_uncore_forcewake_reset (struct drm_device *dev,bool restore); +static inline bool intel_vgpu_active (struct drm_device *dev) +{ +return to_i915 (dev)->vgpu.active; +} +void i915_enable_pipestat(struct
drm_i915_private*dev_priv, enum pipe
pipe,u32 status_mask);drivers/gpu/drm/i915/i915_vgpu.c new file +4
-0@@ -0,0 +1,85 @@ +/* +* Copyright(c) 2011-2014 Intel Corporation. All rights reserved. +* +* Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */drivers/gpu/drm/i915/i915_vgpu.c new file +9
-0+ +#include "intel_drv.h" +#include "i915_vgpu.h" + + +/** +* DOC: Intel GVT-g guest support +* +* Intel GVT-g is a graphics virtualization technology which shares the
+ * GPU among multiple virtual machines on a time-sharing basis. Each
+ * virtual machine is presented a virtual GPU (vGPU), which has equivalent
+ * features as the underlying physical GPU (pGPU), so i915 driver can run
+ * seamlessly in a virtual machine. This file provides vGPU specific
+ * optimizations when running in a virtual machine, to reduce the complexity
+ * of vGPU emulation and to improve the overall performance.
+ *
+ * A primary function introduced here is so-called "address space ballooning"
+ * technique. Intel GVT-g partitions global graphics memory amongQuoted 8 lines
Quoted 5 lines
multiple VMs,
+ * so each VM can directly access a portion of the memory w/o hypervisor's
+ * intervention, e.g. filling textures or queuing commands. However w/ thePersonal preference, but It may be nicer to avoid "w/o" and similar shorthand in documentation.
OK. Will fix it. :)
Quoted 43 lines
Quoted 43 lines
+ * partitioning an unmodified i915 driver would assume a smaller graphics
+ * memory starting from address ZERO, then requires vGPU emulation module to
+ * translate the graphics address between 'guest view' and 'host view', for
+ * all registers and command opcodes which contain a graphics memory address.
+ * To reduce the complexity, Intel GVT-g introduces "address space ballooning",
+ * by telling the exact partitioning knowledge to each guest i915 driver, which
+ * then reserves and prevents non-allocated portions from allocation. Thus vGPU
+ * emulation module module only needs to scan and validate graphics addresses
+ * w/o complexity of address translation.
+ *
+ */drivers/gpu/drm/i915/i915_vgpu.c new file +6
-0+ +/** +* i915_check_vgpu - detect virtual GPU +* @dev: drm device * +* +* This function is called at the initialization stage, to detect whether
+ * running on a vGPU.
+ */drivers/gpu/drm/i915/i915_vgpu.c new file +14
-0+void i915_check_vgpu (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv =to_i915 (dev); +uint64_t magic; +uint32_t version; + +BUILD_BUG_ON (sizeof (struct vgt_if ) !=VGT_PVINFO_SIZE ); + +if (!IS_HASWELL (dev)) +return ; + + magic =readq (dev_priv->regs +vgtif_reg (magic)); +if (magic !=VGT_MAGIC ) +return ;Quoted 8 lines
drivers/gpu/drm/i915/i915_vgpu.c line 81
Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +6
-0+ + version =INTEL_VGT_IF_VERSION_ENCODE ( +readw (dev_priv->regs +vgtif_reg (version_major)), +readw (dev_priv->regs +vgtif_reg (version_minor))); +if (version !=INTEL_VGT_IF_VERSION ) +return ;Would it be useful to something on version mismatch?
Quoted 12 lines
Well. Thanks for your notification. :)
In fact, this version judgement is just for future usages. By now, we have only one vgt interface version,0x10000. But you are right, unlike the judgement for 'magic', which specifies the Intel GVT-g feature, and can be returned directly when false, failure for the 'version' may should trigger something like unload the driver. However, if so, this means we may need to:
1> return something ini915_check_vgpu();
2> change the prototype ofi915_check_vgpu()and its callerintel_uncore_init();
3> check the return value ofintel_uncore_init()ini915_driver_load()Is this approach acceptable?I would leave this for the future version. I was simply suggesting logging something on version mismatch. Just because it is unexpected so there is a trace in the logs why feature is not available. It's up to you.
Quoted 80 lines
Quoted 71 lines
Quoted 71 lines
drivers/gpu/drm/i915/i915_vgpu.c new file +3
-0+ dev_priv->vgpu.active = true; +DRM_INFO ("Virtual GPU for Intel GVT-g detected.\n" ); +}drivers/gpu/drm/i915/i915_vgpu.h new file +4
-0@@ -0,0 +1,85 @@ +/* +* Copyright(c) 2011-2014 Intel Corporation. All rights reserved. +* +* Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */drivers/gpu/drm/i915/i915_vgpu.h new file +5
-0+ +#ifndef _I915_VGPU_H_ +#define _I915_VGPU_H_ + +/* The MMIO offset of the shared info between guest and host emulator */
drivers/gpu/drm/i915/i915_vgpu.h new file +12
-0+#define VGT_PVINFO_PAGE 0x78000 +#define VGT_PVINFO_SIZE 0x1000 + +/* +* The following structure pages are defined in GEN MMIO space +* for virtualization. (One page for now) +*/ +#define VGT_MAGIC 0x4776544776544776/* 'vGTvGTvG' */ +#define VGT_VERSION_MAJOR 1 +#define VGT_VERSION_MINOR 0 + +#define INTEL_VGT_IF_VERSION_ENCODE (major, minor) ((major) << 16 |(minor))
drivers/gpu/drm/i915/i915_vgpu.h new file +5
-0+#define INTEL_VGT_IF_VERSION \ + INTEL_VGT_IF_VERSION_ENCODE(VGT_VERSION_MAJOR, VGT_VERSION_MINOR) + +struct vgt_if { +uint64_t magic;/* VGT_MAGIC */ Quoted 9 lines
drivers/gpu/drm/i915/i915_vgpu.h line 48
Quoted 4 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +4
-0+uint16_t version_major; +uint16_t version_minor; +uint32_t vgt_id;/* ID of vGT instance */ +uint32_t rsv1[12 ];/* pad to offset 0x40 */ In general I thought stdint types should be avoided in the kernel in favour of u32 and company. But I see i915 is full of them already so maybe my information is outdated.
Quoted 1 line
Let's keep the _t types, like explanations in Daniel's mail. Is this OK?
Yes.
Quoted 26 lines
Quoted 26 lines
Quoted 11 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +11
-0+/* +* Data structure to describe the balooning info of resources. +* Each VM can only have one portion of continuous area for now. +* (May support scattered resource in future) +* (starting from offset 0x40) +*/ +struct { +/* Aperture register balooning */ +struct { +uint32_t my_base; +uint32_t my_size;drivers/gpu/drm/i915/i915_vgpu.h line 69
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +10
-0+ } low_gmadr;/* aperture */ +/* GMADR register balooning */ +struct { +uint32_t my_base; +uint32_t my_size; + } high_gmadr;/* non aperture */ +/* allowed fence registers */ +uint32_t fence_num; +uint32_t rsv2[3 ]; + } avail_rs;/* available/assigned resource */ I didn't figure out yet why the concept is to pass in two ranges to mark as reserved, instead of one usable/allowed range and then implicitly all the rest as reserved?
Quoted 6 lines
Quoted 2 lines
Like you said in 2/3 comments, this 2 part is for mappable and non-mapplable respectively. :)
Are high and low memory as terms known in the i915 space so far as synonyms for mappable and non-mappable?
Well, I'm not sure.
Daniel, do you think the "low gmadr/high gmadr" will cause any misunderstanding?
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
drivers/gpu/drm/i915/i915_vgpu.h line 78
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +3
-0+uint32_t rsv3[0x200 -24 ];/* pad to half page */ +/* +* The bottom half page is for response from Gfx driver to hypervisor.
drivers/gpu/drm/i915/i915_vgpu.h new file +6
-0+* Set to reserved fields temporarily by now. +*/ +uint32_t rsv4; +uint32_t display_ready;/* ready for display owner switch */ +uint32_t rsv5[0x200 -2 ];/* pad to one page */ +} __packed;You probably don't need
__packedsince you haveBUILD_BUG_ONon size mismatch and pad to full size with reserved fields. I don't think it matters though.
Quoted 6 lines
Quoted 2 lines
Well, IIRC, this
__packedis added to follow Chris's comments. How about we keep it? Or will it hurt any place? :)The only purpose for packed here I can see is a double guard along the
BUILD_BUG_ON. As I said I don't think it makes any difference so you can leave it in.
OK. Thanks.
Quoted 6 lines
Regards,
Tvrtko
B.R.
Yu
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 1/8] drm/i915: Introduce a PV INFO page structure for Intel GVT-g.
Quoted 32 lines
From: Yu, Zhang
Sent: Wednesday, December 17, 2014 10:50 AMQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/i915/i915_vgpu.h
Quoted 3 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +3
-0+/* +* Data structure to describe the balooning info of resources. +* Each VM can only have one portion of continuous area for now.
Quoted 26 lines
Quoted 25 lines
Quoted 19 lines
Quoted 8 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +8
-0+* (May support scattered resource in future) +* (starting from offset 0x40) +*/ +struct { +/* Aperture register balooning */ +struct { +uint32_t my_base; +uint32_t my_size;drivers/gpu/drm/i915/i915_vgpu.h line 69
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h new file +10
-0+ } low_gmadr;/* aperture */ +/* GMADR register balooning */ +struct { +uint32_t my_base; +uint32_t my_size; + } high_gmadr;/* non aperture */ +/* allowed fence registers */ +uint32_t fence_num; +uint32_t rsv2[3 ]; + } avail_rs;/* available/assigned resource */ Quoted 4 lines
I didn't figure out yet why the concept is to pass in two ranges to mark as reserved, instead of one usable/allowed range and then implicitly all the rest as reserved?
Like you said in 2/3 comments, this 2 part is for mappable and non-mapplable respectively. :)
Quoted 6 lines
Quoted 3 lines
Are high and low memory as terms known in the i915 space so far as synonyms for mappable and non-mappable?
Well, I'm not sure.
Daniel, do you think the "low gmadr/high gmadr" will cause any misunderstanding?
for i915 possibly mappable and non-mappable are more intuitive, but for end user they finally need to know mappable is actually in low end and non-mappable is in high end, so low/high would be simpler to them. since this is the information used between host and guest, we're open to whatever meaningful to i915, and our host vgt driver can still keep low/high notations to user.
Thanks
Kevin
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v3 2/8 drm/i915: Adds graphic address space ballooning logic
In reply to Yu ZhangWith Intel GVT-g, the global graphic memory space is partitioned by multiple vGPU instances in different VMs. The ballooning code is called in i915_gem_setup_global_gtt(), utilizing the drm mm allocator APIs to mark the graphic address space which are partitioned out to other vGPUs as reserved.
v2:
take Chris and Daniel's comments:
- no guard page between different VMs
- use drm_mm_reserve_node() to do the reservation for ballooning, instead of the previous drm_mm_insert_node_in_range_generic()
v3:
take Daniel's comments:
- move ballooning functions into i915_vgpu.c
- add kerneldoc to ballooning functions
--- drivers/gpu/drm/i915/i915_gem_gtt.c | 17 +++- drivers/gpu/drm/i915/i915_vgpu.c | 149 ++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 2 + 3 files changed, 165 insertions(+), 3 deletions(-)
drivers/gpu/drm/i915/i915_gem_gtt.c +14 -3
@@ -27,6 +27,7 @@#include <drm/drmP.h> #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_vgpu.h" #include "i915_trace.h" #include "intel_drv.h"
@@ -1683,6 +1684,16 @@ int i915_gem_setup_global_gtt(struct drm_device *dev,/* Subtract the guard page ... */ drm_mm_init (&ggtt_vm->mm, start, end - start -PAGE_SIZE ); + + dev_priv->gtt.base.start = start; + dev_priv->gtt.base.total = end - start; + +if (intel_vgpu_active (dev)) { + ret =intel_vgt_balloon (dev); +if (ret) +return ret; + } +if (!HAS_LLC (dev)) dev_priv->gtt.base.mm.color_adjust = i915_gtt_color_adjust;
@@ -1702,9 +1713,6 @@ int i915_gem_setup_global_gtt(struct drm_device *dev, vma->bound |=GLOBAL_BIND ; } - dev_priv->gtt.base.start = start; - dev_priv->gtt.base.total = end - start; -/* Clear any non-preallocated blocks */ drm_mm_for_each_hole (entry, &ggtt_vm->mm, hole_start, hole_end) { DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",
@@ -1756,6 +1764,9 @@ void i915_global_gtt_cleanup(struct drm_device *dev) }if (drm_mm_initialized (&vm->mm)) { +if (intel_vgpu_active (dev)) +intel_vgt_deballoon (); +drm_mm_takedown (&vm->mm);list_del (&vm->global_link); }
drivers/gpu/drm/i915/i915_vgpu.c +149 -0
152 lines not on this page · raw message
drivers/gpu/drm/i915/i915_vgpu.h +2 -0
@@ -81,5 +81,7 @@ struct vgt_if { (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)extern void i915_check_vgpu (struct drm_device *dev); +extern int intel_vgt_balloon (struct drm_device *dev); +extern void intel_vgt_deballoon (void );#endif /* _I915_VGPU_H_ */
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 176
Quoted 4 lines · Yu Zhang
drivers/gpu/drm/i915/i915_vgpu.c +4
-0+if (low_gm_base < ggtt_vm->start + || low_gm_end > dev_priv->gtt.mappable_end + || high_gm_base < dev_priv->gtt.mappable_end + || high_gm_end > ggtt_vm_end) {
Nit: Logical operators like || or && should be at the end of the previous line, not at the beginning of the new line. checkpatch would have noticed this one.
-Daniel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic
Quoted 11 lines · Daniel Vetter
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 176
Quoted 4 lines · Yu Zhang
drivers/gpu/drm/i915/i915_vgpu.c +4
-0+if (low_gm_base < ggtt_vm->start + || low_gm_end > dev_priv->gtt.mappable_end + || high_gm_base < dev_priv->gtt.mappable_end + || high_gm_end > ggtt_vm_end) {Nit: Logical operators like || or && should be at the end of the previous line, not at the beginning of the new line. checkpatch would have noticed this one.
-Daniel
Got it.
Thank you. :-)
Yu
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic
In reply to Yu ZhangQuoted 43 lines · Yu Zhang
With Intel GVT-g, the global graphic memory space is partitioned by multiple vGPU instances in different VMs. The ballooning code is called in
i915_gem_setup_global_gtt(), utilizing the drm mm allocator APIs to mark the graphic address space which are partitioned out to other vGPUs as reserved.v2:
take Chris and Daniel's comments:
- no guard page between different VMs
- usedrm_mm_reserve_node()to do the reservation for ballooning, instead of the previousdrm_mm_insert_node_in_range_generic()v3:
take Daniel's comments:
- move ballooning functions intoi915_vgpu.c
- add kerneldoc to ballooning functionsSigned-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Zhi Wang <[email protected]>Signed-off-by Eddie Dong <[email protected]>--- drivers/gpu/drm/i915/i915_gem_gtt.c | 17 +++- drivers/gpu/drm/i915/i915_vgpu.c | 149 ++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 2 + 3 files changed, 165 insertions(+), 3 deletions(-)drivers/gpu/drm/i915/i915_gem_gtt.c +1
-0@@ -27,6 +27,7 @@#include <drm/drmP.h> #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_vgpu.h" #include "i915_trace.h" #include "intel_drv.h" @@ -1683,6 +1684,16 @@ int i915_gem_setup_global_gtt(struct drm_device *dev,/* Subtract the guard page ... */ drm_mm_init (&ggtt_vm->mm, start, end - start -PAGE_SIZE );
drivers/gpu/drm/i915/i915_gem_gtt.c int i915_gem_setup_global_gtt(struct drm_device *dev, · line 1696
Quoted 10 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +10
-0+ + dev_priv->gtt.base.start = start; + dev_priv->gtt.base.total = end - start; + +if (intel_vgpu_active (dev)) { + ret =intel_vgt_balloon (dev); +if (ret) +return ret; + } +
Out of curiosity, what will be the mechanism to prevent a vGPU instance from ignoring the ballooning data? Must be something in the hypervisor blocking pass-through access to such domains?
And probably GPU reset should also be disallowed for vGPU instances?
Quoted 30 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +3
-3if (!HAS_LLC (dev)) dev_priv->gtt.base.mm.color_adjust = i915_gtt_color_adjust;@@ -1702,9 +1713,6 @@ int i915_gem_setup_global_gtt(struct drm_device *dev, vma->bound |=GLOBAL_BIND ; } - dev_priv->gtt.base.start = start; - dev_priv->gtt.base.total = end - start; -/* Clear any non-preallocated blocks */ drm_mm_for_each_hole (entry, &ggtt_vm->mm, hole_start, hole_end) { DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",@@ -1756,6 +1764,9 @@ void i915_global_gtt_cleanup(struct drm_device *dev) }if (drm_mm_initialized (&vm->mm)) { +if (intel_vgpu_active (dev)) +intel_vgt_deballoon (); +drm_mm_takedown (&vm->mm);list_del (&vm->global_link); }drivers/gpu/drm/i915/i915_vgpu.c +0
-0@@ -83,3 +83,152 @@ void i915_check_vgpu(struct drm_device *dev) dev_priv->vgpu.active = true;DRM_INFO ("Virtual GPU for Intel GVT-g detected.\n" );
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 94
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +9
-0} + +struct _balloon_info_ { +/* +* There are up to 2 regions per low/high graphic memory that +* might be ballooned. Here, index 0/1 is for low +* graphic memory, 2/3 for high graphic memory. +*/ +struct drm_mm_node space[4 ]; +} bl_info;
This should be static I think.
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 106
Quoted 11 lines
drivers/gpu/drm/i915/i915_vgpu.c +11
-0+/** +* intel_vgt_deballoon - deballoon reserved graphics address trunks +* +* This function is called to deallocate the ballooned-out graphic memory, when +* driver is unloaded or when ballooning fails. +*/ +void intel_vgt_deballoon (void ) +{ +int i; + +DRM_INFO ("VGT deballoon.\n" );
Would debug be more appropriate? I don't see much value of saying this on driver unload - it's not that it is optional at this point.
Also for all logging, is intended human readable name VGT or vGT? If the latter it would be nicer to log it in that form.
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10
-0+ +for (i =0 ; i <4 ; i++) { +if (bl_info.space[i].allocated) +drm_mm_remove_node (&bl_info.space[i]); + } + +memset (&bl_info,0 ,sizeof (bl_info)); +} + +static int vgt_balloon_space(struct drm_mm *mm,
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 126
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10
-0+struct drm_mm_node *node, + unsigned long start, unsigned long end) +{ + unsigned long size = end - start; + +if (start == end) +return -EINVAL ; + +DRM_INFO ("balloon space: range [ 0x%lx - 0x%lx ] %lu KB.\n" , + start, end, size /1024 );
KiB ?
Quoted 5 lines
drivers/gpu/drm/i915/i915_vgpu.c +5
-0+ node->start = start; + node->size = size; + +return drm_mm_reserve_node (mm, node); +}
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 142
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10
-0+ +/** +* intel_vgt_balloon - balloon out reserved graphics address trunks +* @dev: drm device +* +* This function is called at the initialization stage, to balloon out the +* graphic address space allocated to other VMs, by marking these spaces as +* reserved. +* +* The ballooning related knowledges(starting address and size of the low/high
s/knowledges\(/knowledge /
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10
-0+* graphic memory) are depicted in the vgt_if structure in a reserved MMIO +* range. +* +* Returns: +* zero on success, non-zero if configuration invalid or ballooning failed +*/ +int intel_vgt_balloon (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv =to_i915 (dev); +struct i915_address_space *ggtt_vm = &dev_priv->gtt.base;
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 162
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10
-0+unsigned long ggtt_vm_end = ggtt_vm->start + ggtt_vm->total; + +unsigned long low_gm_base, low_gm_size, low_gm_end; +unsigned long high_gm_base, high_gm_size, high_gm_end; +int ret; + + low_gm_base =I915_READ (vgtif_reg (avail_rs.low_gmadr.my_base)); + low_gm_size =I915_READ (vgtif_reg (avail_rs.low_gmadr.my_size)); + high_gm_base =I915_READ (vgtif_reg (avail_rs.high_gmadr.my_base)); + high_gm_size =I915_READ (vgtif_reg (avail_rs.high_gmadr.my_size));
Get rid of my_ prefix ?
Quoted 9 lines
drivers/gpu/drm/i915/i915_vgpu.c +9
-0+ + low_gm_end = low_gm_base + low_gm_size; + high_gm_end = high_gm_base + high_gm_size; + +DRM_INFO ("VGT ballooning configuration:\n" ); +DRM_INFO ("Low graphic memory: base 0x%lx size %ldKB\n" , + low_gm_base, low_gm_size /1024 ); +DRM_INFO ("High graphic memory: base 0x%lx size %ldKB\n" , + high_gm_base, high_gm_size /1024 );
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 181
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10
-0+ +if (low_gm_base < ggtt_vm->start + || low_gm_end > dev_priv->gtt.mappable_end + || high_gm_base < dev_priv->gtt.mappable_end + || high_gm_end > ggtt_vm_end) { +DRM_ERROR ("Invalid ballooning configuration!\n" ); +return -EINVAL ; + } + +memset (&bl_info,0 ,sizeof (bl_info));
If bl_info is static then you don't need this memset?
Quoted 33 lines
drivers/gpu/drm/i915/i915_vgpu.c +33
-0+/* High graphic memory ballooning */ +if (high_gm_base > dev_priv->gtt.mappable_end) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[2 ], + dev_priv->gtt.mappable_end, + high_gm_base); + +if (ret) + goto err; + } + +/* +* No need to partition out the last physical page, +* because it is reserved to the guard page. +*/ +if (high_gm_end < ggtt_vm_end -PAGE_SIZE ) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[3 ], + high_gm_end, + ggtt_vm_end -PAGE_SIZE ); +if (ret) + goto err; + } + +/* Low graphic memory ballooning */ +if (low_gm_base > ggtt_vm->start) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[0 ], + ggtt_vm->start, low_gm_base); + +if (ret) + goto err; + }
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 225
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10
-0+ +if (low_gm_end < dev_priv->gtt.mappable_end) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[1 ], + low_gm_end, + dev_priv->gtt.mappable_end); + +if (ret) + goto err; + }
Okay, I've figured it out. :) I suppose going back to patch 1, where it says "Each VM can only have one portion of continuous area for now", with the emphasis on _one_. That threw me off thinking you have two "balloons" meaning forbidden areas. And then with low and high naming I got the wrong idea that one balloon marks the bottom inaccessible part, and the other top. I didn't figure out the whole low == mappable, high == non-mappable split. I suppose it was my inexperience, but if you can think of a way on how to improve the comment, even ASCII art would be very nice if possible.
Actually ballooning in the first place confuses me because I thought prior use for that in virtualisation was for memory which can shrink and grow at runtime. Did I get that wrong?
Quoted 14 lines
drivers/gpu/drm/i915/i915_vgpu.c +8
-0+DRM_INFO ("VGT balloon successfully\n" ); +return 0 ; + +err: +DRM_ERROR ("VGT balloon fail\n" ); +intel_vgt_deballoon (); +return ret; +}drivers/gpu/drm/i915/i915_vgpu.h +0
-0
drivers/gpu/drm/i915/i915_vgpu.h struct vgt_if { · line 87
Quoted 7 lines
drivers/gpu/drm/i915/i915_vgpu.h +2
-0@@ -81,5 +81,7 @@ struct vgt_if { (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)extern void i915_check_vgpu (struct drm_device *dev); +extern int intel_vgt_balloon (struct drm_device *dev); +extern void intel_vgt_deballoon (void );#endif /* _I915_VGPU_H_ */
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic
Quoted 46 lines · Tvrtko Ursulin
Quoted 44 lines · Yu Zhang
With Intel GVT-g, the global graphic memory space is partitioned by multiple vGPU instances in different VMs. The ballooning code is called in
i915_gem_setup_global_gtt(), utilizing the drm mm allocator APIs to mark the graphic address space which are partitioned out to other vGPUs as reserved.v2:
take Chris and Daniel's comments:
- no guard page between different VMs
- usedrm_mm_reserve_node()to do the reservation for ballooning, instead of the previousdrm_mm_insert_node_in_range_generic()v3:
take Daniel's comments:
- move ballooning functions intoi915_vgpu.c
- add kerneldoc to ballooning functionsSigned-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Zhi Wang <[email protected]>Signed-off-by Eddie Dong <[email protected]>--- drivers/gpu/drm/i915/i915_gem_gtt.c | 17 +++- drivers/gpu/drm/i915/i915_vgpu.c | 149 ++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 2 + 3 files changed, 165 insertions(+), 3 deletions(-)drivers/gpu/drm/i915/i915_gem_gtt.c +1
-0@@ -27,6 +27,7 @@#include <drm/drmP.h> #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_vgpu.h" #include "i915_trace.h" #include "intel_drv.h" @@ -1683,6 +1684,16 @@ int i915_gem_setup_global_gtt(struct drm_device *dev,
Quoted 16 lines
drivers/gpu/drm/i915/i915_gem_gtt.c int i915_gem_setup_global_gtt(struct drm_device *dev, · line 1696
Quoted 12 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +10
-0/* Subtract the guard page ... */ drm_mm_init (&ggtt_vm->mm, start, end - start -PAGE_SIZE ); + + dev_priv->gtt.base.start = start; + dev_priv->gtt.base.total = end - start; + +if (intel_vgpu_active (dev)) { + ret =intel_vgt_balloon (dev); +if (ret) +return ret; + } +Out of curiosity, what will be the mechanism to prevent a vGPU instance from ignoring the ballooning data? Must be something in the hypervisor blocking pass-through access to such domains?
Well, although we have range check logic in the host side(which checks the legality of a GM address), the correctness of a GM from vGPU side are supposed to be guaranteed by the drm mm allocator - all those ballooned out spaces are marked as reserved.
Quoted 40 lines
And probably GPU reset should also be disallowed for vGPU instances?
Quoted 37 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +0
-3if (!HAS_LLC (dev)) dev_priv->gtt.base.mm.color_adjust = i915_gtt_color_adjust;@@ -1702,9 +1713,6 @@ int i915_gem_setup_global_gtt(struct drm_device *dev, vma->bound |=GLOBAL_BIND ; } - dev_priv->gtt.base.start = start; - dev_priv->gtt.base.total = end - start; -/* Clear any non-preallocated blocks */ drm_mm_for_each_hole (entry, &ggtt_vm->mm, hole_start, hole_end) {DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",
drivers/gpu/drm/i915/i915_gem_gtt.c +3
-0@@ -1756,6 +1764,9 @@ void i915_global_gtt_cleanup(struct drm_device *dev) }if (drm_mm_initialized (&vm->mm)) { +if (intel_vgpu_active (dev)) +intel_vgt_deballoon (); +drm_mm_takedown (&vm->mm);list_del (&vm->global_link);}drivers/gpu/drm/i915/i915_vgpu.c +3
-0@@ -83,3 +83,152 @@ void i915_check_vgpu(struct drm_device *dev) dev_priv->vgpu.active = true;DRM_INFO ("Virtual GPU for Intel GVT-g detected.\n" ); } + +struct _balloon_info_ { +/*
Quoted 8 lines
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 94
Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c +6
-0+* There are up to 2 regions per low/high graphic memory that +* might be ballooned. Here, index 0/1 is for low +* graphic memory, 2/3 for high graphic memory. +*/ +struct drm_mm_node space[4 ]; +} bl_info;This should be static I think.
Yes, you are right.
Quoted 16 lines
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 106
Quoted 13 lines
drivers/gpu/drm/i915/i915_vgpu.c +4
-0+/** +* intel_vgt_deballoon - deballoon reserved graphics address trunks +* +* This function is called to deallocate the ballooned-out graphic memory, when
+ * driver is unloaded or when ballooning fails.
+ */drivers/gpu/drm/i915/i915_vgpu.c +5
-0+void intel_vgt_deballoon (void ) +{ +int i; + +DRM_INFO ("VGT deballoon.\n" );Would debug be more appropriate? I don't see much value of saying this on driver unload - it's not that it is optional at this point.
OK. :)
Quoted 3 lines
Also for all logging, is intended human readable name VGT or vGT? If the latter it would be nicer to log it in that form.
Well, I prefer VGT. :)
Quoted 15 lines
Quoted 15 lines
drivers/gpu/drm/i915/i915_vgpu.c +14
-0+ +for (i =0 ; i <4 ; i++) { +if (bl_info.space[i].allocated) +drm_mm_remove_node (&bl_info.space[i]); + } + +memset (&bl_info,0 ,sizeof (bl_info)); +} + +static int vgt_balloon_space (struct drm_mm *mm, +struct drm_mm_node *node, +unsigned long start,unsigned long end) +{ +unsigned long size = end - start;
Quoted 8 lines
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 126
Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c +6
-0+ +if (start == end) +return -EINVAL ; + +DRM_INFO ("balloon space: range [ 0x%lx - 0x%lx ] %lu KB.\n" , + start, end, size /1024 );KiB ?
Yes. Thanks. :)
Quoted 13 lines
Quoted 13 lines
drivers/gpu/drm/i915/i915_vgpu.c +11
-0+ node->start = start; + node->size = size; + +return drm_mm_reserve_node (mm, node); +} + +/** +* intel_vgt_balloon - balloon out reserved graphics address trunks +* @dev: drm device +* +* This function is called at the initialization stage, to balloon out the
Quoted 8 lines
Quoted 6 lines
+ * graphic address space allocated to other VMs, by marking these spaces as
+ * reserved.
+ *
+ * The ballooning related knowledges(starting address and size of the low/highs/knowledges\(/knowledge /
Got it.
Quoted 17 lines
Quoted 17 lines
+ * graphic memory) are depicted in the
vgt_ifstructure in a reserved MMIO
+ * range.
+ *
+ * Returns:
+ * zero on success, non-zero if configuration invalid or ballooning failed
+ */drivers/gpu/drm/i915/i915_vgpu.c +8
-0+int intel_vgt_balloon (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv =to_i915 (dev); +struct i915_address_space *ggtt_vm = &dev_priv->gtt.base; +unsigned long ggtt_vm_end = ggtt_vm->start + ggtt_vm->total; + +unsigned long low_gm_base, low_gm_size, low_gm_end; +unsigned long high_gm_base, high_gm_size, high_gm_end;
Quoted 8 lines
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 162
Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c +6
-0+int ret; + + low_gm_base =I915_READ (vgtif_reg (avail_rs.low_gmadr.my_base)); + low_gm_size =I915_READ (vgtif_reg (avail_rs.low_gmadr.my_size)); + high_gm_base =I915_READ (vgtif_reg (avail_rs.high_gmadr.my_base)); + high_gm_size =I915_READ (vgtif_reg (avail_rs.high_gmadr.my_size));Get rid of
my_prefix ?
OK. Will do. :)
Quoted 14 lines
Quoted 14 lines
drivers/gpu/drm/i915/i915_vgpu.c +13
-0+ + low_gm_end = low_gm_base + low_gm_size; + high_gm_end = high_gm_base + high_gm_size; + +DRM_INFO ("VGT ballooning configuration:\n" ); +DRM_INFO ("Low graphic memory: base 0x%lx size %ldKB\n" , + low_gm_base, low_gm_size /1024 ); +DRM_INFO ("High graphic memory: base 0x%lx size %ldKB\n" , + high_gm_base, high_gm_size /1024 ); + +if (low_gm_base < ggtt_vm->start + || low_gm_end > dev_priv->gtt.mappable_end + || high_gm_base < dev_priv->gtt.mappable_end
Quoted 8 lines
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 181
Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c +6
-0+ || high_gm_end > ggtt_vm_end) { +DRM_ERROR ("Invalid ballooning configuration!\n" ); +return -EINVAL ; + } + +memset (&bl_info,0 ,sizeof (bl_info));If
bl_infois static then you don't need this memset?
Oh, yes. :)
Quoted 46 lines
Quoted 34 lines
drivers/gpu/drm/i915/i915_vgpu.c +33
-0+/* High graphic memory ballooning */ +if (high_gm_base > dev_priv->gtt.mappable_end) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[2 ], + dev_priv->gtt.mappable_end, + high_gm_base); + +if (ret) + goto err; + } + +/* +* No need to partition out the last physical page, +* because it is reserved to the guard page. +*/ +if (high_gm_end < ggtt_vm_end -PAGE_SIZE ) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[3 ], + high_gm_end, + ggtt_vm_end -PAGE_SIZE ); +if (ret) + goto err; + } + +/* Low graphic memory ballooning */ +if (low_gm_base > ggtt_vm->start) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[0 ], + ggtt_vm->start, low_gm_base); + +if (ret) + goto err; + }drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 225
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10
-0+ +if (low_gm_end < dev_priv->gtt.mappable_end) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[1 ], + low_gm_end, + dev_priv->gtt.mappable_end); + +if (ret) + goto err; + }Okay, I've figured it out. :) I suppose going back to patch 1, where it
Quoted 8 lines
says "Each VM can only have one portion of continuous area for now", with the emphasis on _one_. That threw me off thinking you have two "balloons" meaning forbidden areas. And then with low and high naming I got the wrong idea that one balloon marks the bottom inaccessible part, and the other top. I didn't figure out the whole low == mappable, high == non-mappable split. I suppose it was my inexperience, but if you can think of a way on how to improve the comment, even ASCII art would be very nice if possible.
Hah. Guess I need a better comments above. Let me think how to organize. :)
Quoted 5 lines
Actually ballooning in the first place confuses me because I thought prior use for that in virtualisation was for memory which can shrink and grow at runtime. Did I get that wrong?
Well, for memory virtualization, the word 'ballooning' does mean this - it can shrink and grow.
Yet for our Intel GVT-g, the 'ballooning' is used, to emphasis that we are not just partitioning the GM space to each vgpu, the vgpu can also perceive its GM address space as starting from zero(although the usable GM spaces may probably not). By now, we do not grow/shrink. :)
Quoted 20 lines
Quoted 20 lines
drivers/gpu/drm/i915/i915_vgpu.c +8
-0+DRM_INFO ("VGT balloon successfully\n" ); +return 0 ; + +err: +DRM_ERROR ("VGT balloon fail\n" ); +intel_vgt_deballoon (); +return ret; +}drivers/gpu/drm/i915/i915_vgpu.h +2
-0@@ -81,5 +81,7 @@ struct vgt_if { (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)extern void i915_check_vgpu (struct drm_device *dev); +extern int intel_vgt_balloon (struct drm_device *dev); +extern void intel_vgt_deballoon (void );
Quoted 8 lines
drivers/gpu/drm/i915/i915_vgpu.h struct vgt_if { · line 87
Quoted 2 lines
drivers/gpu/drm/i915/i915_vgpu.h +0
-0#endif /* _I915_VGPU_H_ */ Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic
Quoted 63 lines · Zhang
Quoted 46 lines · Tvrtko Ursulin
Quoted 44 lines · Yu Zhang
With Intel GVT-g, the global graphic memory space is partitioned by multiple vGPU instances in different VMs. The ballooning code is called in
i915_gem_setup_global_gtt(), utilizing the drm mm allocator APIs to mark the graphic address space which are partitioned out to other vGPUs as reserved.v2:
take Chris and Daniel's comments:
- no guard page between different VMs
- usedrm_mm_reserve_node()to do the reservation for ballooning, instead of the previousdrm_mm_insert_node_in_range_generic()v3:
take Daniel's comments:
- move ballooning functions intoi915_vgpu.c
- add kerneldoc to ballooning functionsSigned-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Zhi Wang <[email protected]>Signed-off-by Eddie Dong <[email protected]>--- drivers/gpu/drm/i915/i915_gem_gtt.c | 17 +++- drivers/gpu/drm/i915/i915_vgpu.c | 149 ++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 2 + 3 files changed, 165 insertions(+), 3 deletions(-)drivers/gpu/drm/i915/i915_gem_gtt.c +1
-0@@ -27,6 +27,7 @@#include <drm/drmP.h> #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_vgpu.h" #include "i915_trace.h" #include "intel_drv.h" @@ -1683,6 +1684,16 @@ int i915_gem_setup_global_gtt(struct drm_device *dev,Quoted 16 lines
drivers/gpu/drm/i915/i915_gem_gtt.c int i915_gem_setup_global_gtt(struct drm_device *dev, · line 1696
Quoted 12 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +10
-0/* Subtract the guard page ... */ drm_mm_init (&ggtt_vm->mm, start, end - start -PAGE_SIZE ); + + dev_priv->gtt.base.start = start; + dev_priv->gtt.base.total = end - start; + +if (intel_vgpu_active (dev)) { + ret =intel_vgt_balloon (dev); +if (ret) +return ret; + } +Out of curiosity, what will be the mechanism to prevent a vGPU instance from ignoring the ballooning data? Must be something in the hypervisor blocking pass-through access to such domains?
Quoted 2 lines
Well, although we have range check logic in the host side(which checks the legality of a GM address), the correctness of a GM from vGPU side
Oh, I thought that part is direct for performance reasons (avoiding translation). It's not then?
Quoted 2 lines
are supposed to be guaranteed by the drm mm allocator - all those ballooned out spaces are marked as reserved.
I was thinking about a malicious i915 driver instance, or simply a bug in DRM or i915 which breaks the blocked ranges assumption.
If that is possible then it sounds pretty bad if one VM could snoop the surfaces from another.
Quoted 64 lines
Quoted 39 lines
And probably GPU reset should also be disallowed for vGPU instances?
Quoted 37 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +0
-3if (!HAS_LLC (dev)) dev_priv->gtt.base.mm.color_adjust = i915_gtt_color_adjust;@@ -1702,9 +1713,6 @@ int i915_gem_setup_global_gtt(struct drm_device *dev, vma->bound |=GLOBAL_BIND ; } - dev_priv->gtt.base.start = start; - dev_priv->gtt.base.total = end - start; -/* Clear any non-preallocated blocks */ drm_mm_for_each_hole (entry, &ggtt_vm->mm, hole_start, hole_end) {DRM_DEBUG_KMS("clearing unused GTT space: [%lx, %lx]\n",
drivers/gpu/drm/i915/i915_gem_gtt.c +3
-0@@ -1756,6 +1764,9 @@ void i915_global_gtt_cleanup(struct drm_device *dev) }if (drm_mm_initialized (&vm->mm)) { +if (intel_vgpu_active (dev)) +intel_vgt_deballoon (); +drm_mm_takedown (&vm->mm);list_del (&vm->global_link);}drivers/gpu/drm/i915/i915_vgpu.c +3
-0@@ -83,3 +83,152 @@ void i915_check_vgpu(struct drm_device *dev) dev_priv->vgpu.active = true;DRM_INFO ("Virtual GPU for Intel GVT-g detected.\n" ); } + +struct _balloon_info_ { +/* Quoted 8 lines
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 94
Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c +6
-0+* There are up to 2 regions per low/high graphic memory that +* might be ballooned. Here, index 0/1 is for low +* graphic memory, 2/3 for high graphic memory. +*/ +struct drm_mm_node space[4 ]; +} bl_info;This should be static I think.
Yes, you are right.
Quoted 16 lines
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 106
Quoted 13 lines
drivers/gpu/drm/i915/i915_vgpu.c +4
-0+/** +* intel_vgt_deballoon - deballoon reserved graphics address trunks +* +* This function is called to deallocate the ballooned-out graphic memory, when
+ * driver is unloaded or when ballooning fails.
+ */drivers/gpu/drm/i915/i915_vgpu.c +5
-0+void intel_vgt_deballoon (void ) +{ +int i; + +DRM_INFO ("VGT deballoon.\n" );Would debug be more appropriate? I don't see much value of saying this on driver unload - it's not that it is optional at this point.
Quoted 5 lines
OK. :)
Quoted 3 lines
Also for all logging, is intended human readable name VGT or vGT? If the latter it would be nicer to log it in that form.
Well, I prefer VGT. :)
So it's vGPU and VGT.. OK... :) (and GVT-d) :)
Quoted 148 lines
Quoted 14 lines
Quoted 14 lines
drivers/gpu/drm/i915/i915_vgpu.c +14
-0+ +for (i =0 ; i <4 ; i++) { +if (bl_info.space[i].allocated) +drm_mm_remove_node (&bl_info.space[i]); + } + +memset (&bl_info,0 ,sizeof (bl_info)); +} + +static int vgt_balloon_space (struct drm_mm *mm, +struct drm_mm_node *node, +unsigned long start,unsigned long end) +{ +unsigned long size = end - start;Quoted 8 lines
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 126
Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c +6
-0+ +if (start == end) +return -EINVAL ; + +DRM_INFO ("balloon space: range [ 0x%lx - 0x%lx ] %lu KB.\n" , + start, end, size /1024 );KiB ?
Yes. Thanks. :)
Quoted 13 lines
Quoted 13 lines
drivers/gpu/drm/i915/i915_vgpu.c +11
-0+ node->start = start; + node->size = size; + +return drm_mm_reserve_node (mm, node); +} + +/** +* intel_vgt_balloon - balloon out reserved graphics address trunks +* @dev: drm device +* +* This function is called at the initialization stage, to balloon out the
Quoted 8 lines
Quoted 6 lines
+ * graphic address space allocated to other VMs, by marking these spaces as
+ * reserved.
+ *
+ * The ballooning related knowledges(starting address and size of the low/highs/knowledges\(/knowledge /
Got it.
Quoted 17 lines
Quoted 17 lines
+ * graphic memory) are depicted in the
vgt_ifstructure in a reserved MMIO
+ * range.
+ *
+ * Returns:
+ * zero on success, non-zero if configuration invalid or ballooning failed
+ */drivers/gpu/drm/i915/i915_vgpu.c +8
-0+int intel_vgt_balloon (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv =to_i915 (dev); +struct i915_address_space *ggtt_vm = &dev_priv->gtt.base; +unsigned long ggtt_vm_end = ggtt_vm->start + ggtt_vm->total; + +unsigned long low_gm_base, low_gm_size, low_gm_end; +unsigned long high_gm_base, high_gm_size, high_gm_end;Quoted 8 lines
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 162
Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c +6
-0+int ret; + + low_gm_base =I915_READ (vgtif_reg (avail_rs.low_gmadr.my_base)); + low_gm_size =I915_READ (vgtif_reg (avail_rs.low_gmadr.my_size)); + high_gm_base =I915_READ (vgtif_reg (avail_rs.high_gmadr.my_base)); + high_gm_size =I915_READ (vgtif_reg (avail_rs.high_gmadr.my_size));Get rid of
my_prefix ?OK. Will do. :)
Quoted 14 lines
Quoted 14 lines
drivers/gpu/drm/i915/i915_vgpu.c +13
-0+ + low_gm_end = low_gm_base + low_gm_size; + high_gm_end = high_gm_base + high_gm_size; + +DRM_INFO ("VGT ballooning configuration:\n" ); +DRM_INFO ("Low graphic memory: base 0x%lx size %ldKB\n" , + low_gm_base, low_gm_size /1024 ); +DRM_INFO ("High graphic memory: base 0x%lx size %ldKB\n" , + high_gm_base, high_gm_size /1024 ); + +if (low_gm_base < ggtt_vm->start + || low_gm_end > dev_priv->gtt.mappable_end + || high_gm_base < dev_priv->gtt.mappable_endQuoted 8 lines
drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 181
Quoted 6 lines
drivers/gpu/drm/i915/i915_vgpu.c +6
-0+ || high_gm_end > ggtt_vm_end) { +DRM_ERROR ("Invalid ballooning configuration!\n" ); +return -EINVAL ; + } + +memset (&bl_info,0 ,sizeof (bl_info));If
bl_infois static then you don't need this memset?Oh, yes. :)
Quoted 46 lines
Quoted 34 lines
drivers/gpu/drm/i915/i915_vgpu.c +33
-0+/* High graphic memory ballooning */ +if (high_gm_base > dev_priv->gtt.mappable_end) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[2 ], + dev_priv->gtt.mappable_end, + high_gm_base); + +if (ret) + goto err; + } + +/* +* No need to partition out the last physical page, +* because it is reserved to the guard page. +*/ +if (high_gm_end < ggtt_vm_end -PAGE_SIZE ) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[3 ], + high_gm_end, + ggtt_vm_end -PAGE_SIZE ); +if (ret) + goto err; + } + +/* Low graphic memory ballooning */ +if (low_gm_base > ggtt_vm->start) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[0 ], + ggtt_vm->start, low_gm_base); + +if (ret) + goto err; + }drivers/gpu/drm/i915/i915_vgpu.c void i915_check_vgpu(struct drm_device *dev) · line 225
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.c +10
-0+ +if (low_gm_end < dev_priv->gtt.mappable_end) { + ret =vgt_balloon_space (&ggtt_vm->mm, + &bl_info.space[1 ], + low_gm_end, + dev_priv->gtt.mappable_end); + +if (ret) + goto err; + }Okay, I've figured it out. :) I suppose going back to patch 1, where it
Quoted 8 lines
says "Each VM can only have one portion of continuous area for now", with the emphasis on _one_. That threw me off thinking you have two "balloons" meaning forbidden areas. And then with low and high naming I got the wrong idea that one balloon marks the bottom inaccessible part, and the other top. I didn't figure out the whole low == mappable, high == non-mappable split. I suppose it was my inexperience, but if you can think of a way on how to improve the comment, even ASCII art would be very nice if possible.
Quoted 2 lines
Hah. Guess I need a better comments above. Let me think how to organize. :)
Cool thanks!
Quoted 11 lines
Quoted 4 lines
Actually ballooning in the first place confuses me because I thought prior use for that in virtualisation was for memory which can shrink and grow at runtime. Did I get that wrong?
Well, for memory virtualization, the word 'ballooning' does mean this - it can shrink and grow.
Yet for our Intel GVT-g, the 'ballooning' is used, to emphasis that we are not just partitioning the GM space to each vgpu, the vgpu can also perceive its GM address space as starting from zero(although the usable GM spaces may probably not).
By now, we do not grow/shrink. :)
Unless there are plan to make it grow and shrink it seems that we agree it is confusing. But I suppose it is too late to change the term now?
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic
Hi,
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Out of curiosity, what will be the mechanism to prevent a vGPU instance from ignoring the ballooning data? Must be something in the hypervisor blocking pass-through access to such domains?
Well, although we have range check logic in the host side(which checks the legality of a GM address), the correctness of a GM from vGPU side
Quoted 8 lines
Oh, I thought that part is direct for performance reasons (avoiding translation). It's not then?
Quoted 2 lines
are supposed to be guaranteed by the drm mm allocator - all those ballooned out spaces are marked as reserved.
I was thinking about a malicious i915 driver instance, or simply a bug in DRM or i915 which breaks the blocked ranges assumption.
Cover letter had a link to a longish paper explaining how all this works in detail.
Short summary:
* Direct access is denied by simply mapping only the guests piece of memory into the guest address space.
* Indirect access (via exec buffer commands) is denied by verifying the exec buffer commands (map read-only + verify + hand over to the hardware when checks pass).
* The ballooning basically makes the guest aware what the offset for its piece of memory has, so the references in the execbuffer don't need to be translated, only verified.
Quoted 2 lines
Unless there are plan to make it grow and shrink it seems that we agree it is confusing. But I suppose it is too late to change the term now?
It's confusing indeed (same question came up @ kvm forum after kvmgt presentation). So a better name would be good I think.
cheers,
Gerd
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic
Hi,
Quoted 18 lines · Gerd Hoffmann
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Out of curiosity, what will be the mechanism to prevent a vGPU instance from ignoring the ballooning data? Must be something in the hypervisor blocking pass-through access to such domains?
Well, although we have range check logic in the host side(which checks the legality of a GM address), the correctness of a GM from vGPU side
Quoted 8 lines
Oh, I thought that part is direct for performance reasons (avoiding translation). It's not then?
Quoted 2 lines
are supposed to be guaranteed by the drm mm allocator - all those ballooned out spaces are marked as reserved.
I was thinking about a malicious i915 driver instance, or simply a bug in DRM or i915 which breaks the blocked ranges assumption.
Cover letter had a link to a longish paper explaining how all this works in detail.
Quoted 9 lines
Short summary:
* Direct access is denied by simply mapping only the guests piece of memory into the guest address space.
* Indirect access (via exec buffer commands) is denied by verifying the exec buffer commands (map read-only + verify + hand over to the hardware when checks pass).
* The ballooning basically makes the guest aware what the offset for its piece of memory has, so the references in the execbuffer don't need to be translated, only verified.
Thanks, I did read the blog and powerpoint from the link, but didn't find what you wrote below even after the second read. Probably missed due to lack of domain knowledge.
Anyway, I just wanted to hear that the DRM allocation range is not the only thing preventing access outside the allocated range. :)
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic
Quoted 33 lines · Tvrtko Ursulin
Hi,
Quoted 19 lines · Gerd Hoffmann
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
Out of curiosity, what will be the mechanism to prevent a vGPU instance
from ignoring the ballooning data? Must be something in the hypervisor blocking pass-through access to such domains?Well, although we have range check logic in the host side(which checks the legality of a GM address), the correctness of a GM from vGPU side
Quoted 8 lines
Oh, I thought that part is direct for performance reasons (avoiding translation). It's not then?
Quoted 2 lines
are supposed to be guaranteed by the drm mm allocator - all those ballooned out spaces are marked as reserved.
I was thinking about a malicious i915 driver instance, or simply a bug in DRM or i915 which breaks the blocked ranges assumption.
Cover letter had a link to a longish paper explaining how all this works in detail.
Quoted 9 lines
Short summary:
* Direct access is denied by simply mapping only the guests piece of memory into the guest address space.
* Indirect access (via exec buffer commands) is denied by verifying the exec buffer commands (map read-only + verify + hand over to the hardware when checks pass).
* The ballooning basically makes the guest aware what the offset for its piece of memory has, so the references in the execbuffer don't need to be translated, only verified.
Quoted 6 lines
Thanks, I did read the blog and powerpoint from the link, but didn't find what you wrote below even after the second read. Probably missed due to lack of domain knowledge.
Anyway, I just wanted to hear that the DRM allocation range is not the only thing preventing access outside the allocated range. :)
Thanks, Tvrtko & Gerd.
DRM allocation range is not the only guarantee. We also have range check functions in our host code. MMIO accesses and ring buffer commands will be checked/scanned in the host side to see if there's any illegal GM address.
Quoted 6 lines
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic
Quoted 12 lines
From: Yu, Zhang
Sent: Wednesday, December 17, 2014 11:10 AMQuoted 10 lines · Gerd Hoffmann
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
Out of curiosity, what will be the mechanism to prevent a vGPU instance
from ignoring the ballooning data? Must be something in the hypervisor blocking pass-through access to such domains?Well, although we have range check logic in the host side(which checks the legality of a GM address), the correctness of a GM from vGPU side
Oh, I thought that part is direct for performance reasons (avoiding translation). It's not then?
yes, just for performance reason to avoid translation
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
are supposed to be guaranteed by the drm mm allocator - all those ballooned out spaces are marked as reserved.
I was thinking about a malicious i915 driver instance, or simply a bug in DRM or i915 which breaks the blocked ranges assumption.
host will validate all the addresses submitted by the guest, w/ the assumption that ballooning is in place so if the address is outside of the allocated machine graphics memory address, it's caught and prevented. We use ballooning to let guest handle the correctness itself, but having Xen to ensure isolation.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Cover letter had a link to a longish paper explaining how all this works in detail.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Short summary:
* Direct access is denied by simply mapping only the guests piece of memory into the guest address space.
* Indirect access (via exec buffer commands) is denied by verifying the exec buffer commands (map read-only + verify + hand over to the hardware when checks pass).
* The ballooning basically makes the guest aware what the offset for its piece of memory has, so the references in the execbuffer don't need to be translated, only verified.
correct.
Quoted 5 lines
Quoted 5 lines
Thanks, I did read the blog and powerpoint from the link, but didn't find what you wrote below even after the second read. Probably missed due to lack of domain knowledge.
Quoted 8 lines
Quoted 2 lines
Anyway, I just wanted to hear that the DRM allocation range is not the only thing preventing access outside the allocated range. :)
Thanks, Tvrtko & Gerd.
DRM allocation range is not the only guarantee. We also have range check functions in our host code. MMIO accesses and ring buffer commands will be checked/scanned in the host side to see if there's any illegal GM address.
and if you read the paper referred in 1st mail, we call our ballooning as address space ballooning which is different from traditional ballooning technique used in memory virtualization, w/ the major difference as:
* traditional ballooning only care about the size. As long as the desired number of pages can be ballooned out, we don't care whether they actually come from.
* address space ballooning used in our case care about both size and address. We need the guest to balloon out a specified range of addresses as explained earlier.
whether to grow/shrink dynamic is not the key difference between two approaches. yes, we only support static ballooning now at boot time, but the same interface can be extended to support dynamic ballooning in the future, w/ more cooperation from guest driver in the future.
So I think using ballooning is a right fit here, though the policy is very simple now.
Thanks
Kevin
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic
Quoted 12 lines · Kevin
and if you read the paper referred in 1st mail, we call our ballooning as address space ballooning which is different from traditional ballooning technique used in memory virtualization, w/ the major difference as:
* traditional ballooning only care about the size. As long as the desired number of pages can be ballooned out, we don't care whether they actually come from.
* address space ballooning used in our case care about both size and address. We need the guest to balloon out a specified range of addresses as explained earlier.
Quoted 7 lines
whether to grow/shrink dynamic is not the key difference between two approaches. yes, we only support static ballooning now at boot time, but the same interface can be extended to support dynamic ballooning in the future, w/ more cooperation from guest driver in the future.
So I think using ballooning is a right fit here, though the policy is very simple now.
Personally I think the first part of the argument does not hold since ballooning is the key word here - address space or memory makes little difference to the metaphor. It is more like partitioning of blanking as it stands.
You only get away with it since you hint of plans to make it dynamic in the future. :)
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 2/8] drm/i915: Adds graphic address space ballooning logic
In reply to Tvrtko UrsulinQuoted 22 lines
From: Tvrtko Ursulin
Sent: Friday, December 12, 2014 9:00 PMReview int i915_gem_setup_global_gtt(struct
Quoted 1 line
(quoted diff) +0
-0@@ -1683,6 +1684,16 @@ int i915_gem_setup_global_gtt(struct
drm_device*dev,drivers/gpu/drm/i915/i915_gem_gtt.c int i915_gem_setup_global_gtt(struct drm_device *dev, · line 1696
Quoted 13 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +10
-0/* Subtract the guard page ... */ drm_mm_init (&ggtt_vm->mm, start, end - start -PAGE_SIZE ); + + dev_priv->gtt.base.start = start; + dev_priv->gtt.base.total = end - start; + +if (intel_vgpu_active (dev)) { + ret =intel_vgt_balloon (dev); +if (ret) +return ret; + } +Out of curiosity, what will be the mechanism to prevent a vGPU instance from ignoring the ballooning data? Must be something in the hypervisor blocking pass-through access to such domains?
Quoted 2 lines
And probably GPU reset should also be disallowed for vGPU instances?
GPU reset is allowed and the host side needs to emulate the vGPU reset behavior. A simple example is that we need remove this vGPU from the scheduling queue, upon a vGPU reset request.
Thanks
Kevin
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v3 3/8 drm/i915: Partition the fence registers for vGPU in i915 driver
In reply to Yu ZhangWith Intel GVT-g, the fence registers are partitioned by multiple vGPU instances in different VMs. Routine i915_gem_load() is modified to reset the num_fence_regs, when the driver detects it's running in a VM. And the allocated fence number is provided in PV INFO page structure.
--- drivers/gpu/drm/i915/i915_gem.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +5 -0
@@ -29,6 +29,7 @@#include <drm/drm_vma_manager.h> #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_vgpu.h" #include "i915_trace.h" #include "intel_drv.h" #include <linux/oom.h>
@@ -5014,6 +5015,10 @@ i915_gem_load(struct drm_device *dev)else dev_priv->num_fence_regs =8 ; +if (intel_vgpu_active (dev)) + dev_priv->num_fence_regs = +I915_READ (vgtif_reg (avail_rs.fence_num)); +/* Initialize fence registers to zero */ INIT_LIST_HEAD (&dev_priv->mm.fence_list);i915_gem_restore_fences (dev);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver
Quoted 27 lines · Yu Zhang
With Intel GVT-g, the fence registers are partitioned by multiple vGPU instances in different VMs. Routine
i915_gem_load()is modified to reset thenum_fence_regs, when the driver detects it's running in a VM. And the allocated fence number is provided in PV INFO page structure.Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Eddie Dong <[email protected]>--- drivers/gpu/drm/i915/i915_gem.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/gpu/drm/i915/i915_gem.c +1
-0@@ -29,6 +29,7 @@#include <drm/drm_vma_manager.h> #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_vgpu.h" #include "i915_trace.h" #include "intel_drv.h" #include <linux/oom.h>
drivers/gpu/drm/i915/i915_gem.c i915_gem_load(struct drm_device *dev) · line 5024
Quoted 10 lines
drivers/gpu/drm/i915/i915_gem.c +4
-0@@ -5014,6 +5015,10 @@ i915_gem_load(struct drm_device *dev)else dev_priv->num_fence_regs =8 ; +if (intel_vgpu_active (dev)) + dev_priv->num_fence_regs = +I915_READ (vgtif_reg (avail_rs.fence_num)); +/* Initialize fence registers to zero */ INIT_LIST_HEAD (&dev_priv->mm.fence_list);i915_gem_restore_fences (dev);
You don't need a start offset and number of allocated fences per domain?
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver
Quoted 33 lines · Tvrtko Ursulin
Quoted 30 lines · Yu Zhang
With Intel GVT-g, the fence registers are partitioned by multiple vGPU instances in different VMs. Routine
i915_gem_load()is modified to reset thenum_fence_regs, when the driver detects it's running in a VM. And the allocated fence number is provided in PV INFO page structure.Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Eddie Dong <[email protected]>--- drivers/gpu/drm/i915/i915_gem.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/gpu/drm/i915/i915_gem.c +1
-0@@ -29,6 +29,7 @@#include <drm/drm_vma_manager.h> #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_vgpu.h" #include "i915_trace.h" #include "intel_drv.h" #include <linux/oom.h> @@ -5014,6 +5015,10 @@ i915_gem_load(struct drm_device *dev)else dev_priv->num_fence_regs =8 ;
Quoted 10 lines
drivers/gpu/drm/i915/i915_gem.c i915_gem_load(struct drm_device *dev) · line 5024
Quoted 8 lines
drivers/gpu/drm/i915/i915_gem.c +4
-0+if (intel_vgpu_active (dev)) + dev_priv->num_fence_regs = +I915_READ (vgtif_reg (avail_rs.fence_num)); +/* Initialize fence registers to zero */ INIT_LIST_HEAD (&dev_priv->mm.fence_list);i915_gem_restore_fences (dev);You don't need a start offset and number of allocated fences per domain?
The PV INFO structure is shared between each vgpu and host, so I guess this is per domain? Not sure if I get your exact meaning. :)
Quoted 3 lines
Regards,
B.R.
Yu
Quoted 3 lines
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver
Quoted 42 lines · Zhang
Quoted 31 lines · Tvrtko Ursulin
Quoted 30 lines · Yu Zhang
With Intel GVT-g, the fence registers are partitioned by multiple vGPU instances in different VMs. Routine
i915_gem_load()is modified to reset thenum_fence_regs, when the driver detects it's running in a VM. And the allocated fence number is provided in PV INFO page structure.Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Eddie Dong <[email protected]>--- drivers/gpu/drm/i915/i915_gem.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/gpu/drm/i915/i915_gem.c +1
-0@@ -29,6 +29,7 @@#include <drm/drm_vma_manager.h> #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_vgpu.h" #include "i915_trace.h" #include "intel_drv.h" #include <linux/oom.h> @@ -5014,6 +5015,10 @@ i915_gem_load(struct drm_device *dev)else dev_priv->num_fence_regs =8 ;Quoted 10 lines
drivers/gpu/drm/i915/i915_gem.c i915_gem_load(struct drm_device *dev) · line 5024
Quoted 8 lines
drivers/gpu/drm/i915/i915_gem.c +4
-0+if (intel_vgpu_active (dev)) + dev_priv->num_fence_regs = +I915_READ (vgtif_reg (avail_rs.fence_num)); +/* Initialize fence registers to zero */ INIT_LIST_HEAD (&dev_priv->mm.fence_list);i915_gem_restore_fences (dev);You don't need a start offset and number of allocated fences per domain?
Quoted 3 lines
The PV INFO structure is shared between each vgpu and host, so I guess this is per domain?
Not sure if I get your exact meaning. :)
I didn't figure out how each domain knowns which fences to use? They know how many, but which ones?
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver
Hi,
Quoted 2 lines
I didn't figure out how each domain knowns which fences to use? They know how many, but which ones?
I think the guest doesn't have to know because mmio access is trapped by the hypervisor anyway.
cheers,
Gerd
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver
Hi,
Quoted 5 lines · Gerd Hoffmann
Quoted 2 lines
I didn't figure out how each domain knowns which fences to use? They know how many, but which ones?
I think the guest doesn't have to know because mmio access is trapped by the hypervisor anyway.
Okay.. so the point is to ensure no domain can hog all fences, but if the hypervisor is already tracking everything why is this limiting required then in the guest? Hypervisor could just say all fences are allocated since it knows how much it gave out per guest.
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver
Quoted 6 lines · Tvrtko Ursulin
Hi,
Quoted 3 lines · Gerd Hoffmann
Quoted 2 lines
I didn't figure out how each domain knowns which fences to use? They know how many, but which ones?
Quoted 7 lines
Quoted 2 lines
I think the guest doesn't have to know because mmio access is trapped by the hypervisor anyway.
Okay.. so the point is to ensure no domain can hog all fences, but if the hypervisor is already tracking everything why is this limiting required then in the guest? Hypervisor could just say all fences are allocated since it knows how much it gave out per guest.
There is no dynamic allocation. Each guest gets a fixed number of fences assigned which it can use, pretty much like memory where each guest gets a fixed piece too.
It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.
cheers,
Gerd
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver
Quoted 19 lines · Gerd Hoffmann
Quoted 6 lines · Tvrtko Ursulin
Hi,
Quoted 3 lines · Gerd Hoffmann
Quoted 2 lines
I didn't figure out how each domain knowns which fences to use? They know how many, but which ones?
Quoted 7 lines
Quoted 2 lines
I think the guest doesn't have to know because mmio access is trapped by the hypervisor anyway.
Okay.. so the point is to ensure no domain can hog all fences, but if the hypervisor is already tracking everything why is this limiting required then in the guest? Hypervisor could just say all fences are allocated since it knows how much it gave out per guest.
There is no dynamic allocation. Each guest gets a fixed number of fences assigned which it can use, pretty much like memory where each guest gets a fixed piece too.
Quoted 5 lines
It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.
That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how many?
With this scheme it could be one guest wants more and can't get them even if no one else is using any. If not limited they could be dynamically allocated by the hypervisor, no?
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver
Hi,
Quoted 8 lines
Quoted 5 lines
It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.
That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how many?
There is a fixed assignment of fences to guests, so it's a fixed number. But as the hypervisor is involved in any fence access anyway there is no need for the guest to know which of the fences it owns, the hypervisor can remap that transparently for the guest, without performance penalty.
Quoted 3 lines
With this scheme it could be one guest wants more and can't get them even if no one else is using any. If not limited they could be dynamically allocated by the hypervisor, no?
Should be possible as extension, but I think[1] for now the goal is to stay as close as possible to physical hardware, where you can't dynamically allocate fences in the first place.
cheers,
Gerd
[1] Not being deeply involved into vgpu development, just reviewing the bits with some kvm background.
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver
Quoted 8 lines · Gerd Hoffmann
Hi,Quoted 6 lines
Quoted 5 lines
It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.
Quoted 7 lines
Quoted 2 lines
That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how many?
There is a fixed assignment of fences to guests, so it's a fixed number. But as the hypervisor is involved in any fence access anyway there is no need for the guest to know which of the fences it owns, the hypervisor can remap that transparently for the guest, without performance penalty.
Thanks Gerd. Exactly.
Although fence registers are parititioned to vGPU, it is not necessary for a vGPU to know the physical mmio addresses of the allocated fence registers.
For example, vGPU 1 with fence size 4 can access the fence registers from 0x100000-10001f; at the same time, vGPU 2 with fence size 8 can access the fence registers from 0x100000-0x10003f. Although this seems conflicting, it does not matter. Because these mmio addresses are all supposed to be trapped in the host side, which will keep a record of the real fence offset of different vGPUs(say 0 for vGPU 1 and 4 for vGPU 2), and then does the remapping. Therefore, the physical operations on the fence register will be performed by host code on different ones(say, 0x100000-10001fh for vGPU 1 and 0x100020-0x10005f for vGPU 2).
Quoted 8 lines
Quoted 4 lines
With this scheme it could be one guest wants more and can't get them even if no one else is using any. If not limited they could be dynamically allocated by the hypervisor, no?
Should be possible as extension, but I think[1] for now the goal is to stay as close as possible to physical hardware, where you can't dynamically allocate fences in the first place.
Yes. By now we have feature to configure the fence numbers for a vGPU. But we do not support the dynamical allocations.
Quoted 10 lines
cheers,
Gerd[1] Not being deeply involved into vgpu development, just reviewing the bits with some kvm background.
Thanks
Yu
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver
Quoted 16 lines · Zhang
Quoted 8 lines · Gerd Hoffmann
Hi,Quoted 6 lines
Quoted 5 lines
It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.
Quoted 7 lines
Quoted 2 lines
That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how many?
There is a fixed assignment of fences to guests, so it's a fixed number. But as the hypervisor is involved in any fence access anyway there is no need for the guest to know which of the fences it owns, the hypervisor can remap that transparently for the guest, without performance penalty.
Quoted 13 lines
Thanks Gerd. Exactly.
Although fence registers are parititioned to vGPU, it is not necessary for a vGPU to know the physical mmio addresses of the allocated fence registers.
For example, vGPU 1 with fence size 4 can access the fence registers from 0x100000-10001f; at the same time, vGPU 2 with fence size 8 can access the fence registers from 0x100000-0x10003f. Although this seems conflicting, it does not matter. Because these mmio addresses are all supposed to be trapped in the host side, which will keep a record of the real fence offset of different vGPUs(say 0 for vGPU 1 and 4 for vGPU 2), and then does the remapping. Therefore, the physical operations on the fence register will be performed by host code on differentones(say, 0x100000-10001fh for vGPU 1 and 0x100020-0x10005f for vGPU 2).
Okay, I think I get it now. What I had in mind is not really possible without a dedicated hypervisor<->guest communication channel. Or in other words you would have to extend the way i915 allocates them from mmio writes to something bi-directional.
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver
Quoted 32 lines · Tvrtko Ursulin
Quoted 16 lines · Zhang
Quoted 8 lines · Gerd Hoffmann
Hi,
Quoted 6 lines
Quoted 5 lines
It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.
Quoted 7 lines
Quoted 2 lines
That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how many?
There is a fixed assignment of fences to guests, so it's a fixed number. But as the hypervisor is involved in any fence access anyway there is no need for the guest to know which of the fences it owns, the hypervisor can remap that transparently for the guest, without performance penalty.
Quoted 13 lines
Thanks Gerd. Exactly.
Although fence registers are parititioned to vGPU, it is not necessary for a vGPU to know the physical mmio addresses of the allocated fence registers.
For example, vGPU 1 with fence size 4 can access the fence registers from 0x100000-10001f; at the same time, vGPU 2 with fence size 8 can access the fence registers from 0x100000-0x10003f. Although this seems conflicting, it does not matter. Because these mmio addresses are all supposed to be trapped in the host side, which will keep a record of the real fence offset of different vGPUs(say 0 for vGPU 1 and 4 for vGPU 2), and then does the remapping. Therefore, the physical operations on the fence register will be performed by host code on differentones(say, 0x100000-10001fh for vGPU 1 and 0x100020-0x10005f for vGPU 2).
Quoted 4 lines
Okay, I think I get it now. What I had in mind is not really possible without a dedicated hypervisor<->guest communication channel. Or in other words you would have to extend the way i915 allocates them from mmio writes to something bi-directional.
You could virtualize fences the same way we virtualize fences for userspace for gtt mmap access: If we need to steal a fences we simply need to unmap the relevant gtt mmio range from the guest ptes. This should work well since on current platforms the only thing that really needs fences is cpu access, the gpu doesn't need them. Well except for some oddball cases in the display block, but those are virtualized anyway (not fbc for guests or anything else like that).
This would also fit a bit more closely with how the host manages fences, so benefiting the new kvm/xengt-on-i915 mode for the host instead of the current implementation which also virtualizes host i915 access cycles. -Daniel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver
Quoted 46 lines · Daniel Vetter
Quoted 32 lines · Tvrtko Ursulin
Quoted 16 lines · Zhang
Quoted 8 lines · Gerd Hoffmann
Hi,
Quoted 6 lines
Quoted 5 lines
It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.
Quoted 7 lines
Quoted 2 lines
That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how many?
There is a fixed assignment of fences to guests, so it's a fixed number. But as the hypervisor is involved in any fence access anyway there is no need for the guest to know which of the fences it owns, the hypervisor can remap that transparently for the guest, without performance penalty.
Quoted 13 lines
Thanks Gerd. Exactly.
Although fence registers are parititioned to vGPU, it is not necessary for a vGPU to know the physical mmio addresses of the allocated fence registers.
For example, vGPU 1 with fence size 4 can access the fence registers from 0x100000-10001f; at the same time, vGPU 2 with fence size 8 can access the fence registers from 0x100000-0x10003f. Although this seems conflicting, it does not matter. Because these mmio addresses are all supposed to be trapped in the host side, which will keep a record of the real fence offset of different vGPUs(say 0 for vGPU 1 and 4 for vGPU 2), and then does the remapping. Therefore, the physical operations on the fence register will be performed by host code on differentones(say, 0x100000-10001fh for vGPU 1 and 0x100020-0x10005f for vGPU 2).Quoted 4 lines
Okay, I think I get it now. What I had in mind is not really possible without a dedicated hypervisor<->guest communication channel. Or in other words you would have to extend the way i915 allocates them from mmio writes to something bi-directional.
You could virtualize fences the same way we virtualize fences for userspace for gtt mmap access: If we need to steal a fences we simply need to unmap the relevant gtt mmio range from the guest ptes. This should work well since on current platforms the only thing that really needs fences is cpu access, the gpu doesn't need them. Well except for some oddball cases in the display block, but those are virtualized anyway (not fbc for guests or anything else like that).
Quoted 3 lines
This would also fit a bit more closely with how the host manages fences, so benefiting the new kvm/xengt-on-i915 mode for the host instead of the current implementation which also virtualizes host i915 access cycles.
Btw this isn't a blocker for the current implementation since we can always just tell guests that they can use all fences when we implement this. So the current, simpler implementation isn't restricting us in any meaningful way.
-Daniel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver
In reply to Daniel VetterQuoted 42 lines
From: Daniel Vetter
Sent: Thursday, December 18, 2014 1:10 AMQuoted 32 lines · Tvrtko Ursulin
Quoted 16 lines · Zhang
Quoted 8 lines · Gerd Hoffmann
Hi,
Quoted 6 lines
Quoted 5 lines
It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.
Quoted 7 lines
Quoted 2 lines
That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how many?
There is a fixed assignment of fences to guests, so it's a fixed number. But as the hypervisor is involved in any fence access anyway there is no need for the guest to know which of the fences it owns, the hypervisor can remap that transparently for the guest, without performance penalty.
Quoted 13 lines
Thanks Gerd. Exactly.
Although fence registers are parititioned to vGPU, it is not necessary for a vGPU to know the physical mmio addresses of the allocated fence registers.
For example, vGPU 1 with fence size 4 can access the fence registers from 0x100000-10001f; at the same time, vGPU 2 with fence size 8 can access the fence registers from 0x100000-0x10003f. Although this seems conflicting, it does not matter. Because these mmio addresses are all supposed to be trapped in the host side, which will keep a record of the real fence offset of different vGPUs(say 0 for vGPU 1 and 4 for vGPU 2), and then does the remapping. Therefore, the physical operations on the fence register will be performed by host code on differentones(say, 0x100000-10001fh for vGPU 1 and 0x100020-0x10005f for vGPU 2).Quoted 3 lines
Okay, I think I get it now. What I had in mind is not really possible without a dedicated hypervisor<->guest communication channel. Or in other words you would have to extend the way i915 allocates them from mmio
writes
Quoted 1 line
to something bi-directional.
Quoted 7 lines
You could virtualize fences the same way we virtualize fences for userspace for gtt mmap access: If we need to steal a fences we simply need to unmap the relevant gtt mmio range from the guest ptes. This should work well since on current platforms the only thing that really needs fences is cpu access, the gpu doesn't need them. Well except for some oddball cases in the display block, but those are virtualized anyway (not fbc for guests or anything else like that).
doing this w/o guest's awareness is challenging. bear in mind that vCPU scheduling (in hypervisor) is split from vGPU scheduling (in xengt driver). so xengt driver doesn't know when cpu access happens, and thus steal a fence under the hood is problematic.
based on current enlightenment framework, it's possible to have driver allocate/release fences through PVINFO window. That way we can avoid the current static partitioning scheme, and the changes shouldn't be that much.
btw curious I heard that typically 2-4 fences are OK for most workloads. Is that true? If yes, this fence partition along is not the only limiting factor on #VM. the 512M aperture size is more limiting (but will be better starting from bdw)
Quoted 5 lines
This would also fit a bit more closely with how the host manages fences, so benefiting the new kvm/xengt-on-i915 mode for the host instead of the current implementation which also virtualizes host i915 access cycles. -Daniel
for host yes we won't virtualize fence then. We're evaluating how to remove most mediations on the host paths, with few exceptions e.g. for registers which requires manual save/restore at context swtich time.
Thanks
Kevin
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver
Quoted 50 lines · Kevin
Quoted 42 lines
From: Daniel Vetter
Sent: Thursday, December 18, 2014 1:10 AMQuoted 32 lines · Tvrtko Ursulin
Quoted 16 lines · Zhang
Quoted 8 lines · Gerd Hoffmann
Hi,
Quoted 6 lines
Quoted 5 lines
It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the hypervisor anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.
Quoted 7 lines
Quoted 2 lines
That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how many?
There is a fixed assignment of fences to guests, so it's a fixed number. But as the hypervisor is involved in any fence access anyway there is no need for the guest to know which of the fences it owns, the hypervisor can remap that transparently for the guest, without performance penalty.
Quoted 13 lines
Thanks Gerd. Exactly.
Although fence registers are parititioned to vGPU, it is not necessary for a vGPU to know the physical mmio addresses of the allocated fence registers.
For example, vGPU 1 with fence size 4 can access the fence registers from 0x100000-10001f; at the same time, vGPU 2 with fence size 8 can access the fence registers from 0x100000-0x10003f. Although this seems conflicting, it does not matter. Because these mmio addresses are all supposed to be trapped in the host side, which will keep a record of the real fence offset of different vGPUs(say 0 for vGPU 1 and 4 for vGPU 2), and then does the remapping. Therefore, the physical operations on the fence register will be performed by host code on differentones(say, 0x100000-10001fh for vGPU 1 and 0x100020-0x10005f for vGPU 2).Quoted 3 lines
Okay, I think I get it now. What I had in mind is not really possible without a dedicated hypervisor<->guest communication channel. Or in other words you would have to extend the way i915 allocates them from mmio
writes
Quoted 1 line
to something bi-directional.
Quoted 7 lines
You could virtualize fences the same way we virtualize fences for userspace for gtt mmap access: If we need to steal a fences we simply need to unmap the relevant gtt mmio range from the guest ptes. This should work well since on current platforms the only thing that really needs fences is cpu access, the gpu doesn't need them. Well except for some oddball cases in the display block, but those are virtualized anyway (not fbc for guests or anything else like that).
Quoted 4 lines
doing this w/o guest's awareness is challenging. bear in mind that vCPU scheduling (in hypervisor) is split from vGPU scheduling (in xengt driver). so xengt driver doesn't know when cpu access happens, and thus steal a fence under the hood is problematic.
Well i915 also doesn't interfere with the linux scheduler at all to manage fences. All you have to do is kick out the ptes before yanking the fence away, no need to talk to the scheduler at all. Ofc if some other vm is using that area it'll immediately fault on the cpu, but the fault handler can then rectify this and steal some other fence that's not used right away again hopefully.
Quoted 16 lines
based on current enlightenment framework, it's possible to have driver allocate/release fences through PVINFO window. That way we can avoid the current static partitioning scheme, and the changes shouldn't be that much.
btw curious I heard that typically 2-4 fences are OK for most workloads. Is that true? If yes, this fence partition along is not the only limiting factor on #VM. the 512M aperture size is more limiting (but will be better starting from bdw)
Quoted 5 lines
This would also fit a bit more closely with how the host manages fences, so benefiting the new kvm/xengt-on-i915 mode for the host instead of the current implementation which also virtualizes host i915 access cycles. -Daniel
Quoted 4 lines
for host yes we won't virtualize fence then. We're evaluating how to remove most mediations on the host paths, with few exceptions e.g. for registers which requires manual save/restore at context swtich time.
My idea would be to use additional MI_LRI/SRM commands before/after the call to load the context into the ring using the functions in i915_gem_context.c. This is exactly what the execbuf code does too:
There's lots of additional stuff (per-platform) it does around the actual context switch. So even there I think you can implement the VM switch on top of i915 infrastructure. Maybe we need to duplicate/export some execbuf internals, but that's a lot better than duplicating the entire driver and having 2 drivers fight over the same hw.
-Daniel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver
Quoted 67 lines
From: Daniel Vetter [mailto:[email protected]] On Behalf Of Daniel Vetter
Sent: Thursday, December 18, 2014 4:08 PMQuoted 11 lines · Kevin
Quoted 11 lines
From: Daniel Vetter
Sent: Thursday, December 18, 2014 1:10 AMQuoted 7 lines · Tvrtko Ursulin
Quoted 5 lines · Zhang
Quoted 4 lines · Gerd Hoffmann
Hi,
Quoted 2 lines
Quoted 2 lines
It's not possible to allow guests direct access to the fence registers though. And if every fence register access traps into the
hypervisor
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
anyway the hypervisor can easily map the guest virtual fence to host physical fence, so there is no need to tell the guest which fences it owns, the number of fences is enough.
That exactly is the part I don't understand - if it is not required to tell the guest which fences it owns, why it is required to say how
many?
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
There is a fixed assignment of fences to guests, so it's a fixed number. But as the hypervisor is involved in any fence access anyway there is
no
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
need for the guest to know which of the fences it owns, the hypervisor can remap that transparently for the guest, without performance
penalty.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Thanks Gerd. Exactly.
Although fence registers are parititioned to vGPU, it is not necessary for a vGPU to know the physical mmio addresses of the allocated fence registers.
For example, vGPU 1 with fence size 4 can access the fence registers from 0x100000-10001f; at the same time, vGPU 2 with fence size 8 can access the fence registers from 0x100000-0x10003f. Although thisseems
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
conflicting, it does not matter. Because these mmio addresses are all supposed to be trapped in the host side, which will keep a record of the real fence offset of different vGPUs(say 0 for vGPU 1 and 4 for vGPU 2), and then does the remapping. Therefore, the physical operations on the fence register will be performed by host code on different
ones(say, 0x100000-10001fh for vGPU 1 and 0x100020-0x10005f for vGPU 2).Okay, I think I get it now. What I had in mind is not really possible without a dedicated hypervisor<->guest communication channel. Or in
other
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
words you would have to extend the way i915 allocates them from mmio
writes
Quoted 1 line
to something bi-directional.
You could virtualize fences the same way we virtualize fences for userspace for gtt mmap access: If we need to steal a fences we simply
need
Quoted 1 line
Quoted 1 line
to unmap the relevant gtt mmio range from the guest ptes. This should
work
Quoted 9 lines
Quoted 4 lines
well since on current platforms the only thing that really needs fences is cpu access, the gpu doesn't need them. Well except for some oddball cases in the display block, but those are virtualized anyway (not fbc for guests or anything else like that).
doing this w/o guest's awareness is challenging. bear in mind that vCPU scheduling (in hypervisor) is split from vGPU scheduling (in xengt driver). so xengt driver doesn't know when cpu access happens, and thus steal a fence under the hood is problematic.
Quoted 6 lines
Well i915 also doesn't interfere with the linux scheduler at all to manage fences. All you have to do is kick out the ptes before yanking the fence away, no need to talk to the scheduler at all. Ofc if some other vm is using that area it'll immediately fault on the cpu, but the fault handler can then rectify this and steal some other fence that's not used right away again hopefully.
yes, this would work, but I'd think having guest to allocate/release fence through PVINFO is simpler and less hypervisor specific.
Quoted 22 lines
Quoted 17 lines
based on current enlightenment framework, it's possible to have driver allocate/release fences through PVINFO window. That way we can avoid the current static partitioning scheme, and the changes shouldn't be that much.
btw curious I heard that typically 2-4 fences are OK for most workloads. Is that true? If yes, this fence partition along is not the only limiting factor on #VM. the 512M aperture size is more limiting (but will be better starting from bdw)
Quoted 5 lines
This would also fit a bit more closely with how the host manages fences, so benefiting the new kvm/xengt-on-i915 mode for the host instead of the current implementation which also virtualizes host i915 access cycles. -Daniel
Quoted 4 lines
for host yes we won't virtualize fence then. We're evaluating how to remove most mediations on the host paths, with few exceptions e.g. for registers which requires manual save/restore at context swtich time.
Quoted 10 lines
My idea would be to use additional MI_LRI/SRM commands before/after the call to load the context into the ring using the functions in
i915_gem_context.c. This is exactly what the execbuf code does too:
There's lots of additional stuff (per-platform) it does around the actual context switch. So even there I think you can implement the VM switch on top of i915 infrastructure. Maybe we need to duplicate/export some execbuf internals, but that's a lot better than duplicating the entire driver and having 2 drivers fight over the same hw.
-Daniel
--
yes, that's the direction what we agreed before and definitely we don't want to duplicate entire driver (btw even with current i915-on-xengt model our driver code is much smaller since we only focus on resource management. :-)) We'll try to leverage existing i915 functions as many as possible. For major changes, we'll summarize and send out for design review to reach agreement. for above example, yes what you described should work, by hooking xengt specific register list in 915_gem_context.c
Thanks
Kevin
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 3/8] drm/i915: Partition the fence registers for vGPU in i915 driver
In reply to Tvrtko UrsulinQuoted 47 lines
From: Tvrtko Ursulin
Sent: Tuesday, December 16, 2014 9:44 PMQuoted 42 lines · Zhang
Quoted 41 lines · Tvrtko Ursulin
Quoted 28 lines · Yu Zhang
With Intel GVT-g, the fence registers are partitioned by multiple vGPU instances in different VMs. Routine
i915_gem_load()is modified to reset thenum_fence_regs, when the driver detects it's running in a VM. And the allocated fence number is provided in PV INFO page structure.Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Eddie Dong <[email protected]>--- drivers/gpu/drm/i915/i915_gem.c | 5 +++++ 1 file changed, 5 insertions(+)drivers/gpu/drm/i915/i915_gem.c +1
-0@@ -29,6 +29,7 @@#include <drm/drm_vma_manager.h> #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_vgpu.h" #include "i915_trace.h" #include "intel_drv.h" #include <linux/oom.h> drivers/gpu/drm/i915/i915_gem.c i915_gem_load(struct drm_device *dev) · line 5024
Quoted 10 lines
drivers/gpu/drm/i915/i915_gem.c +4
-0@@ -5014,6 +5015,10 @@ i915_gem_load(struct drm_device *dev)else dev_priv->num_fence_regs =8 ; +if (intel_vgpu_active (dev)) + dev_priv->num_fence_regs = +I915_READ (vgtif_reg (avail_rs.fence_num)); +/* Initialize fence registers to zero */ INIT_LIST_HEAD (&dev_priv->mm.fence_list);i915_gem_restore_fences (dev);You don't need a start offset and number of allocated fences per domain?
Quoted 7 lines
Quoted 3 lines
The PV INFO structure is shared between each vgpu and host, so I guess this is per domain?
Not sure if I get your exact meaning. :)I didn't figure out how each domain knowns which fences to use? They know how many, but which ones?
the offset is maintained in the host side, which will be added to guest index in relevant emulation path.
Thanks
Kevin
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v3 4/8 drm/i915: Disable framebuffer compression for i915 driver in VM
In reply to Yu ZhangFramebuffer compression is disabled when driver detects it's running in a Intel GVT-g enlightened VM, because FBC is not emulated and there is no stolen memory for a vGPU.
v2:
take Chris' comments:
- move the code into intel_update_fbc()
--- drivers/gpu/drm/i915/intel_pm.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/intel_pm.c +4 -0
@@ -544,6 +544,10 @@ void intel_update_fbc(struct drm_device *dev)return ; } +/* disable framebuffer compression in vGPU */ +if (intel_vgpu_active (dev)) + i915.enable_fbc =0 ; +/* * If FBC is already on, we just have to verify that we can * keep it that way...
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 4/8] drm/i915: Disable framebuffer compression for i915 driver in VM
Quoted 21 lines · Yu Zhang
Framebuffer compression is disabled when driver detects it's running in a Intel GVT-g enlightened VM, because FBC is not emulated and there is no stolen memory for a vGPU.
v2:
take Chris' comments:
- move the code intointel_update_fbc()Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Zhiyuan Lv <[email protected]>--- drivers/gpu/drm/i915/intel_pm.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/gpu/drm/i915/intel_pm.c +0
-0
drivers/gpu/drm/i915/intel_pm.c void intel_update_fbc(struct drm_device *dev) · line 553
Quoted 10 lines
drivers/gpu/drm/i915/intel_pm.c +4
-0@@ -544,6 +544,10 @@ void intel_update_fbc(struct drm_device *dev)return ; } +/* disable framebuffer compression in vGPU */ +if (intel_vgpu_active (dev)) + i915.enable_fbc =0 ; +/* * If FBC is already on, we just have to verify that we can * keep it that way...
Looks like you'll need to rebase this one, I see no intel_update_fbc in my tree. :(
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 4/8] drm/i915: Disable framebuffer compression for i915 driver in VM
Quoted 27 lines · Tvrtko Ursulin
Quoted 24 lines · Yu Zhang
Framebuffer compression is disabled when driver detects it's running in a Intel GVT-g enlightened VM, because FBC is not emulated and there is no stolen memory for a vGPU.
v2:
take Chris' comments:
- move the code intointel_update_fbc()Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Zhiyuan Lv <[email protected]>--- drivers/gpu/drm/i915/intel_pm.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/gpu/drm/i915/intel_pm.c +0
-0@@ -544,6 +544,10 @@ void intel_update_fbc(struct drm_device *dev)return ; }
Quoted 12 lines
drivers/gpu/drm/i915/intel_pm.c void intel_update_fbc(struct drm_device *dev) · line 553
Quoted 8 lines
drivers/gpu/drm/i915/intel_pm.c +4
-0+/* disable framebuffer compression in vGPU */ +if (intel_vgpu_active (dev)) + i915.enable_fbc =0 ; +/* * If FBC is already on, we just have to verify that we can * keep it that way... Looks like you'll need to rebase this one, I see no
intel_update_fbcin my tree. :(
OK. I'll rebase this in next version. :)
Quoted 5 lines
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v3 5/8 drm/i915: Add the display switch logic for vGPU in i915 driver
In reply to Yu ZhangDisplay switch logic is added to notify the host side that current vGPU have a valid surface to show. It does so by writing the display_ready field in PV INFO page, and then will be handled in the host side. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host side.
v2:
- move the notification code outside the 'else' in load sequence
- remove the notification code in intel_crtc_set_config()
--- drivers/gpu/drm/i915/i915_dma.c | 11 +++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 7 +++++++ 2 files changed, 18 insertions(+)
drivers/gpu/drm/i915/i915_dma.c +11 -0
@@ -36,6 +36,7 @@#include "intel_drv.h" #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_vgpu.h" #include "i915_trace.h" #include <linux/pci.h> #include <linux/console.h>
@@ -1780,6 +1781,16 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) dev_priv->ums.mm_suspended =1 ; } +if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +struct drm_i915_private *dev_priv =to_i915 (dev); + +I915_WRITE (vgtif_reg (display_ready),VGT_DRV_DISPLAY_READY ); + } +i915_setup_sysfs (dev);if (INTEL_INFO (dev)->num_pipes) {
drivers/gpu/drm/i915/i915_vgpu.h +7 -0
@@ -80,6 +80,13 @@ struct vgt_if {#define vgtif_reg (x) \ (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x) +/* vGPU display status to be used by the host side */ +enum vgt_display_status { +VGT_DRV_DISPLAY_NOT_READY =0 , +VGT_DRV_DISPLAY_READY ,/* ready for display switch */ +}; + +extern void i915_check_vgpu (struct drm_device *dev);extern int intel_vgt_balloon (struct drm_device *dev);extern void intel_vgt_deballoon (void );
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 5/8] drm/i915: Add the display switch logic for vGPU in i915 driver
Quoted 36 lines · Yu Zhang
Display switch logic is added to notify the host side that current vGPU have a valid surface to show. It does so by writing the
display_readyfield in PV INFO page, and then will be handled in the host side. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host side.v2:
- move the notification code outside the 'else' in load sequence
- remove the notification code inintel_crtc_set_config()Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Zhiyuan Lv <[email protected]>--- drivers/gpu/drm/i915/i915_dma.c | 11 +++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 7 +++++++ 2 files changed, 18 insertions(+)drivers/gpu/drm/i915/i915_dma.c +1
-0@@ -36,6 +36,7 @@#include "intel_drv.h" #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_vgpu.h" #include "i915_trace.h" #include <linux/pci.h> #include <linux/console.h> @@ -1780,6 +1781,16 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) dev_priv->ums.mm_suspended =1 ; }
drivers/gpu/drm/i915/i915_dma.c int i915_driver_load(struct drm_device *dev, unsigned long flags) · line 1793
Quoted 10 lines
drivers/gpu/drm/i915/i915_dma.c +10
-0+if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +struct drm_i915_private *dev_priv =to_i915 (dev); + +I915_WRITE (vgtif_reg (display_ready),VGT_DRV_DISPLAY_READY ); + } +
Current code base already has dev_priv so looks like another rebase. Otherwise pretty simple so:
Quoted 12 lines
drivers/gpu/drm/i915/i915_dma.c +0
-0i915_setup_sysfs (dev);if (INTEL_INFO (dev)->num_pipes) {drivers/gpu/drm/i915/i915_vgpu.h +0
-0@@ -80,6 +80,13 @@ struct vgt_if {#define vgtif_reg (x) \ (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)
drivers/gpu/drm/i915/i915_vgpu.h struct vgt_if { · line 92
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h +7
-0+/* vGPU display status to be used by the host side */ +enum vgt_display_status { +VGT_DRV_DISPLAY_NOT_READY =0 , +VGT_DRV_DISPLAY_READY ,/* ready for display switch */ +}; + +extern void i915_check_vgpu (struct drm_device *dev);extern int intel_vgt_balloon (struct drm_device *dev);extern void intel_vgt_deballoon (void );
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 5/8] drm/i915: Add the display switch logic for vGPU in i915 driver
In reply to Yu ZhangQuoted 54 lines · Yu Zhang
Display switch logic is added to notify the host side that current vGPU have a valid surface to show. It does so by writing the
display_readyfield in PV INFO page, and then will be handled in the host side. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host side.v2:
- move the notification code outside the 'else' in load sequence
- remove the notification code inintel_crtc_set_config()Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Zhiyuan Lv <[email protected]>--- drivers/gpu/drm/i915/i915_dma.c | 11 +++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 7 +++++++ 2 files changed, 18 insertions(+)drivers/gpu/drm/i915/i915_dma.c +11
-0@@ -36,6 +36,7 @@#include "intel_drv.h" #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_vgpu.h" #include "i915_trace.h" #include <linux/pci.h> #include <linux/console.h> @@ -1780,6 +1781,16 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) dev_priv->ums.mm_suspended =1 ; } +if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +struct drm_i915_private *dev_priv =to_i915 (dev); + +I915_WRITE (vgtif_reg (display_ready),VGT_DRV_DISPLAY_READY ); + } +i915_setup_sysfs (dev);if (INTEL_INFO (dev)->num_pipes) {drivers/gpu/drm/i915/i915_vgpu.h +0
-0
drivers/gpu/drm/i915/i915_vgpu.h struct vgt_if { · line 87
Quoted 8 lines
drivers/gpu/drm/i915/i915_vgpu.h +5
-0@@ -80,6 +80,13 @@ struct vgt_if {#define vgtif_reg (x) \ (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x) +/* vGPU display status to be used by the host side */ +enum vgt_display_status { +VGT_DRV_DISPLAY_NOT_READY =0 , +VGT_DRV_DISPLAY_READY ,/* ready for display switch */ +};
Nitpick: Since this is a virtual register I prefer we use plain #defines with explicit codes for everything instead of enums. Makes it clearer which values are used and all that.
-Daniel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 5/8] drm/i915: Add the display switch logic for vGPU in i915 driver
Quoted 58 lines · Daniel Vetter
Quoted 57 lines · Yu Zhang
Display switch logic is added to notify the host side that current vGPU have a valid surface to show. It does so by writing the
display_readyfield in PV INFO page, and then will be handled in the host side. This is useful to avoid trickiness when the VM's framebuffer is being accessed in the middle of VM modesetting, e.g. compositing the framebuffer in the host side.v2:
- move the notification code outside the 'else' in load sequence
- remove the notification code inintel_crtc_set_config()Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Zhiyuan Lv <[email protected]>--- drivers/gpu/drm/i915/i915_dma.c | 11 +++++++++++ drivers/gpu/drm/i915/i915_vgpu.h | 7 +++++++ 2 files changed, 18 insertions(+)drivers/gpu/drm/i915/i915_dma.c +11
-0@@ -36,6 +36,7 @@#include "intel_drv.h" #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_vgpu.h" #include "i915_trace.h" #include <linux/pci.h> #include <linux/console.h> @@ -1780,6 +1781,16 @@ int i915_driver_load(struct drm_device *dev, unsigned long flags) dev_priv->ums.mm_suspended =1 ; } +if (intel_vgpu_active (dev)) { +/* +* Notify a valid surface after modesetting, +* when running inside a VM. +*/ +struct drm_i915_private *dev_priv =to_i915 (dev); + +I915_WRITE (vgtif_reg (display_ready),VGT_DRV_DISPLAY_READY ); + } +i915_setup_sysfs (dev);if (INTEL_INFO (dev)->num_pipes) {drivers/gpu/drm/i915/i915_vgpu.h +0
-0@@ -80,6 +80,13 @@ struct vgt_if {#define vgtif_reg (x) \ (VGT_PVINFO_PAGE + (long)&((struct vgt_if *) NULL)->x)
Quoted 10 lines
drivers/gpu/drm/i915/i915_vgpu.h struct vgt_if { · line 87
Quoted 5 lines
drivers/gpu/drm/i915/i915_vgpu.h +5
-0+/* vGPU display status to be used by the host side */ +enum vgt_display_status { +VGT_DRV_DISPLAY_NOT_READY =0 , +VGT_DRV_DISPLAY_READY ,/* ready for display switch */ +};Nitpick: Since this is a virtual register I prefer we use plain #defines with explicit codes for everything instead of enums. Makes it clearer which values are used and all that.
-Daniel
Got it. #define will be used. And I will rebase this patch.
Quoted 1 line
Thanks
Yu
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v3 6/8 drm/i915: Disable power management for i915 driver in VM
In reply to Yu ZhangWith Intel GVT-g, GPU power management is controlled by host driver, so there is no need to provide virtualized GPU PM support. In the future it might be useful to gather VM input for freq boost, but now let's disable it simply.
v2:
take Chris' comments:
- do not special case this to gen6+
--- drivers/gpu/drm/i915/intel_pm.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/intel_pm.c +4 -0
@@ -5314,6 +5314,10 @@ void intel_enable_gt_powersave(struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private; +/* Powersaving is controlled by the host when inside a VM */ +if (intel_vgpu_active (dev)) +return ; +if (IS_IRONLAKE_M (dev)) {mutex_lock (&dev->struct_mutex);ironlake_enable_drps (dev);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 6/8] drm/i915: Disable power management for i915 driver in VM
Quoted 21 lines · Yu Zhang
With Intel GVT-g, GPU power management is controlled by host driver, so there is no need to provide virtualized GPU PM support. In the future it might be useful to gather VM input for freq boost, but now let's disable it simply.
v2:
take Chris' comments:
- do not special case this to gen6+Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>--- drivers/gpu/drm/i915/intel_pm.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/gpu/drm/i915/intel_pm.c +0
-0
drivers/gpu/drm/i915/intel_pm.c void intel_enable_gt_powersave(struct drm_device *dev) · line 5323
Quoted 10 lines
drivers/gpu/drm/i915/intel_pm.c +4
-0@@ -5314,6 +5314,10 @@ void intel_enable_gt_powersave(struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private; +/* Powersaving is controlled by the host when inside a VM */ +if (intel_vgpu_active (dev)) +return ; +if (IS_IRONLAKE_M (dev)) {mutex_lock (&dev->struct_mutex);ironlake_enable_drps (dev);
I was looking for potential call sites of this which come earlier than i915_check_vgpu. Didn't find any but found RPS (intel_pm_setup) - what's with that - should it be disabled as well?
Otherwise,
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 6/8] drm/i915: Disable power management for i915 driver in VM
Quoted 28 lines · Tvrtko Ursulin
Quoted 25 lines · Yu Zhang
With Intel GVT-g, GPU power management is controlled by host driver, so there is no need to provide virtualized GPU PM support. In the future it might be useful to gather VM input for freq boost, but now let's disable it simply.
v2:
take Chris' comments:
- do not special case this to gen6+Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>--- drivers/gpu/drm/i915/intel_pm.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/gpu/drm/i915/intel_pm.c +0
-0@@ -5314,6 +5314,10 @@ void intel_enable_gt_powersave(struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private;
Quoted 13 lines
Quoted 8 lines
drivers/gpu/drm/i915/intel_pm.c +4
-0+/* Powersaving is controlled by the host when inside a VM */ +if (intel_vgpu_active (dev)) +return ; +if (IS_IRONLAKE_M (dev)) {mutex_lock (&dev->struct_mutex);ironlake_enable_drps(dev);I was looking for potential call sites of this which come earlier than
i915_check_vgpu. Didn't find any but found RPS (intel_pm_setup) - what's with that - should it be disabled as well?
IIUC, the intel_pm_setup() only add the intel_gen6_powersave_work() to the delayed_resume_work queue. Real call of this routine will be delayed to intel_enable_gt_powersave(). So I guess no necessary to do anything in intel_pm_setup(). :)
Quoted 4 lines
Otherwise,
Reviewed-by Tvrtko Ursulin <[email protected]>
B.R.
Yu
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v3 7/8 drm/i915: Create vGPU specific write MMIO to reduce traps
In reply to Yu ZhangIn the virtualized environment, forcewake operations are not necessory for the driver, because mmio accesses will be trapped and emulated by the host side, and real forcewake operations are also done in the host. New mmio write handlers are added to directly call the __raw_i915_write, therefore will reduce many traps and increase the overall performance for drivers running in the VM with Intel GVT-g enhancement.
v2:
take Chris' comments:
- register the mmio hooks in intel_uncore_init()
v3:
take Daniel's comments:
- use macros to assign mmio write functions for vGPU
--- drivers/gpu/drm/i915/intel_uncore.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
drivers/gpu/drm/i915/intel_uncore.c +16 -0
@@ -727,6 +727,14 @@ hsw_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace)REG_WRITE_FOOTER ; \ } +#define __vgpu_write (x) \ +static void \ +vgpu_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \ + REG_WRITE_HEADER; \ + __raw_i915_write##x(dev_priv, reg, val); \ + REG_WRITE_FOOTER; \ +} +static const u32 gen8_shadowed_regs[] = {FORCEWAKE_MT ,GEN6_RPNSWREQ ,
@@ -821,6 +829,10 @@ __gen4_write(8) __gen4_write(16) __gen4_write(32) __gen4_write(64) +__vgpu_write(8) +__vgpu_write(16) +__vgpu_write(32) +__vgpu_write(64)#undef __chv_write#undef __gen8_write
@@ -828,6 +840,7 @@ __gen4_write(64)#undef __gen6_write#undef __gen5_write#undef __gen4_write +#undef __vgpu_write#undef REG_WRITE_FOOTER#undef REG_WRITE_HEADER
@@ -939,6 +952,9 @@ void intel_uncore_init(struct drm_device *dev)break ; } +if (intel_vgpu_active (dev)) +ASSIGN_WRITE_MMIO_VFUNCS (vgpu); +i915_check_and_clear_faults (dev); }#undef ASSIGN_WRITE_MMIO_VFUNCS
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 7/8] drm/i915: Create vGPU specific write MMIO to reduce traps
Quoted 2 lines · Yu Zhang
In the virtualized environment, forcewake operations are not necessory for the driver, because mmio accesses will be trapped
necessary
Quoted 60 lines
and emulated by the host side, and real forcewake operations are also done in the host. New mmio write handlers are added to directly call the
__raw_i915_write, therefore will reduce many traps and increase the overall performance for drivers running in the VM with Intel GVT-g enhancement.v2:
take Chris' comments:
- register the mmio hooks inintel_uncore_init()
v3:
take Daniel's comments:
- use macros to assign mmio write functions for vGPUSigned-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Kevin Tian <[email protected]>--- drivers/gpu/drm/i915/intel_uncore.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)drivers/gpu/drm/i915/intel_uncore.c +13
-0@@ -727,6 +727,14 @@ hsw_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace)REG_WRITE_FOOTER ; \ } +#define __vgpu_write (x) \ +static void \ +vgpu_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \ + REG_WRITE_HEADER; \ + __raw_i915_write##x(dev_priv, reg, val); \ + REG_WRITE_FOOTER; \ +} +static const u32 gen8_shadowed_regs[] = {FORCEWAKE_MT ,GEN6_RPNSWREQ ,@@ -821,6 +829,10 @@ __gen4_write(8) __gen4_write(16) __gen4_write(32) __gen4_write(64) +__vgpu_write(8) +__vgpu_write(16) +__vgpu_write(32) +__vgpu_write(64)#undef __chv_write#undef __gen8_write@@ -828,6 +840,7 @@ __gen4_write(64)#undef __gen6_write#undef __gen5_write#undef __gen4_write +#undef __vgpu_write#undef REG_WRITE_FOOTER#undef REG_WRITE_HEADER
drivers/gpu/drm/i915/intel_uncore.c void intel_uncore_init(struct drm_device *dev) · line 960
Quoted 9 lines
drivers/gpu/drm/i915/intel_uncore.c +3
-0@@ -939,6 +952,9 @@ void intel_uncore_init(struct drm_device *dev)break ; } +if (intel_vgpu_active (dev)) +ASSIGN_WRITE_MMIO_VFUNCS (vgpu); +i915_check_and_clear_faults (dev); }#undef ASSIGN_WRITE_MMIO_VFUNCS
Maybe I am missing something obvious, but why are read variants not needed?
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 7/8] drm/i915: Create vGPU specific write MMIO to reduce traps
Quoted 6 lines · Tvrtko Ursulin
Quoted 2 lines · Yu Zhang
In the virtualized environment, forcewake operations are not necessory for the driver, because mmio accesses will be trapped
necessary
Thanks.
Quoted 66 lines
Quoted 66 lines
and emulated by the host side, and real forcewake operations are also done in the host. New mmio write handlers are added to directly call the
__raw_i915_write, therefore will reduce many traps and increase the overall performance for drivers running in the VM with Intel GVT-g enhancement.v2:
take Chris' comments:
- register the mmio hooks inintel_uncore_init()
v3:
take Daniel's comments:
- use macros to assign mmio write functions for vGPUSigned-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>Signed-off-by Kevin Tian <[email protected]>--- drivers/gpu/drm/i915/intel_uncore.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)drivers/gpu/drm/i915/intel_uncore.c +8
-0@@ -727,6 +727,14 @@ hsw_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace)REG_WRITE_FOOTER ; \ } +#define __vgpu_write (x) \ +static void \ +vgpu_write##x(struct drm_i915_private *dev_priv, off_t reg, u##x val, bool trace) { \ +REG_WRITE_HEADER ; \ + __raw_i915_write##x(dev_priv, reg, val); \ + REG_WRITE_FOOTER; \ +} + static const u32 gen8_shadowed_regs[] = {
FORCEWAKE_MT,
GEN6_RPNSWREQ,drivers/gpu/drm/i915/intel_uncore.c +5
-0@@ -821,6 +829,10 @@ __gen4_write(8) __gen4_write(16) __gen4_write(32) __gen4_write(64) +__vgpu_write(8) +__vgpu_write(16) +__vgpu_write(32) +__vgpu_write(64)#undef __chv_write#undef __gen8_write@@ -828,6 +840,7 @@ __gen4_write(64)#undef __gen6_write#undef __gen5_write#undef __gen4_write +#undef __vgpu_write#undef REG_WRITE_FOOTER#undef REG_WRITE_HEADER@@ -939,6 +952,9 @@ void intel_uncore_init(struct drm_device *dev)break ; }
Quoted 9 lines
drivers/gpu/drm/i915/intel_uncore.c void intel_uncore_init(struct drm_device *dev) · line 960
Quoted 7 lines
drivers/gpu/drm/i915/intel_uncore.c +3
-0+if (intel_vgpu_active (dev)) +ASSIGN_WRITE_MMIO_VFUNCS (vgpu); +i915_check_and_clear_faults (dev); }#undef ASSIGN_WRITE_MMIO_VFUNCSMaybe I am missing something obvious, but why are read variants not needed?
Wah. You got me. Guess I missed this during rebase. Thank you, Tvrtko.
Quoted 6 lines
Regards,
Tvrtko
B.R.
Yu
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v3 8/8 drm/i915: Support alias ppgtt in VM if ppgtt is enabled
In reply to Yu ZhangThe current Intel GVT-g only supports alias ppgtt. And the emulation is done in the host by first trapping PP_DIR_BASE mmio accesses. Updating PP_DIR_BASE by using instructions such as MI_LOAD_REGISTER_IMM are hard to detect and are not supported in current code. Therefore this patch also added a new callback routine - vgpu_mm_switch() to set the PP_DIR_BASE by mmio writes.
v2:
take Chris' comments:
- move the code into sanitize_enable_ppgtt()
--- drivers/gpu/drm/i915/i915_gem_gtt.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
drivers/gpu/drm/i915/i915_gem_gtt.c +16 -0
@@ -44,6 +44,9 @@ static int sanitize_enable_ppgtt(struct drm_device *dev, int enable_ppgtt)if (IS_GEN8 (dev)) has_full_ppgtt = false;/* XXX why? */ +if (intel_vgpu_active (dev)) + has_full_ppgtt = false;/* emulation is too hard */ +if (enable_ppgtt ==0 || !has_aliasing_ppgtt)return 0 ;
@@ -733,6 +736,16 @@ static int hsw_mm_switch(struct i915_hw_ppgtt *ppgtt,return 0 ; } +static int vgpu_mm_switch (struct i915_hw_ppgtt *ppgtt, +struct intel_engine_cs *ring) +{ +struct drm_i915_private *dev_priv =to_i915 (ppgtt->base.dev); + +I915_WRITE (RING_PP_DIR_DCLV (ring),PP_DIR_DCLV_2G ); +I915_WRITE (RING_PP_DIR_BASE (ring),get_pd_offset (ppgtt)); +return 0 ; +} +static int gen7_mm_switch (struct i915_hw_ppgtt *ppgtt,struct intel_engine_cs *ring) {
@@ -1059,6 +1072,9 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt) }else BUG (); +if (intel_vgpu_active (dev)) + ppgtt->switch_mm = vgpu_mm_switch; + ret =gen6_ppgtt_alloc (ppgtt);if (ret)return ret;
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 8/8] drm/i915: Support alias ppgtt in VM if
-------------------------------------Summary-------------------------------------
Platform: baseline_drm_intel_nightly_pass_rate->patch_applied_pass_rate
BYT: pass/total=291/291->290/291
PNV: pass/total=356/356->356/356
ILK: pass/total=372/372->371/372
IVB: pass/total=544/546->544/546
SNB: pass/total=423/425->424/425
HSW: pass/total=579/579->579/579
BDW: pass/total=421/423->422/423
-------------------------------------Detailed-------------------------------------test_platform: test_suite, test_case, result_with_drm_intel_nightly(count, machine_id...)...->result_with_patch_applied(count, machine_id)...
BYT: Intel_gpu_tools, igt_kms_setmode_invalid-clone-single-crtc, TIMEOUT(15, M36M31)PASS(1, M38) -> TIMEOUT(4, M31)
ILK: Intel_gpu_tools, igt_kms_setmode_invalid-clone-single-crtc, FAIL(3, M26)DMESG_FAIL(1, M26)TIMEOUT(24, M37M6M26)PASS(1, M26) -> TIMEOUT(4, M37)
IVB: Intel_gpu_tools, igt_gem_bad_reloc_negative-reloc, NSPT(5, M4M21M34M45)PASS(8, M21M34M4M45) -> NSPT(1, M34)PASS(3, M34)
IVB: Intel_gpu_tools, igt_kms_pipe_crc_basic_hang-read-crc-pipe-B, PASS(4, M4M34) -> TIMEOUT(1, M34)PASS(3, M34)
SNB: Intel_gpu_tools, igt_gem_concurrent_blit_gpuX-bcs-overwrite-source-forked, FAIL(1, M22)PASS(9, M35M22) -> FAIL(1, M22)PASS(3, M22)
BDW: Intel_gpu_tools, igt_gem_reset_stats_ban-bsd, DMESG_WARN(1, M30)PASS(9, M30M42) -> PASS(4, M42)
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v3 8/8] drm/i915: Support alias ppgtt in VM if ppgtt is enabled
In reply to Yu ZhangQuoted 50 lines · Yu Zhang
The current Intel GVT-g only supports alias ppgtt. And the emulation is done in the host by first trapping
PP_DIR_BASEmmio accesses. UpdatingPP_DIR_BASEby using instructions such asMI_LOAD_REGISTER_IMMare hard to detect and are not supported in current code. Therefore this patch also added a new callback routine -vgpu_mm_switch()to set thePP_DIR_BASEby mmio writes.v2:
take Chris' comments:
- move the code intosanitize_enable_ppgtt()Signed-off-by Yu Zhang <[email protected]>Signed-off-by Jike Song <[email protected]>--- drivers/gpu/drm/i915/i915_gem_gtt.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)drivers/gpu/drm/i915/i915_gem_gtt.c +13
-0@@ -44,6 +44,9 @@ static int sanitize_enable_ppgtt(struct drm_device *dev, int enable_ppgtt)if (IS_GEN8 (dev)) has_full_ppgtt = false;/* XXX why? */ +if (intel_vgpu_active (dev)) + has_full_ppgtt = false;/* emulation is too hard */ +if (enable_ppgtt ==0 || !has_aliasing_ppgtt)return 0 ;@@ -733,6 +736,16 @@ static int hsw_mm_switch(struct i915_hw_ppgtt *ppgtt,return 0 ; } +static int vgpu_mm_switch (struct i915_hw_ppgtt *ppgtt, +struct intel_engine_cs *ring) +{ +struct drm_i915_private *dev_priv =to_i915 (ppgtt->base.dev); + +I915_WRITE (RING_PP_DIR_DCLV (ring),PP_DIR_DCLV_2G ); +I915_WRITE (RING_PP_DIR_BASE (ring),get_pd_offset (ppgtt)); +return 0 ; +} +static int gen7_mm_switch (struct i915_hw_ppgtt *ppgtt,struct intel_engine_cs *ring) {
drivers/gpu/drm/i915/i915_gem_gtt.c static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt) · line 1080
Quoted 9 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +3
-0@@ -1059,6 +1072,9 @@ static int gen6_ppgtt_init(struct i915_hw_ppgtt *ppgtt) }else BUG (); +if (intel_vgpu_active (dev)) + ppgtt->switch_mm = vgpu_mm_switch; + ret =gen6_ppgtt_alloc (ppgtt);if (ret)return ret;
Pending potential "checkpatch.pl --strict" fixes,
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 45 lines · Yu Zhang
Intel GVT-g (previously known as XenGT), is a complete GPU virtualization solution with mediated pass-through for 4th generation Intel Core processors - Haswell platform. This technology presents a virtual full-fledged GPU to each Virtual Machine (VM). VMs can directly access performance-critical resources, without intervention from the hypervisor in most cases, while privileged operations from VMs are trap-and-emulated at minimal cost. For detail, please refer to
https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-releaseThis patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, will be sent out in another patchset.
The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory w/o hypervisor's intervention, e.g. filling textures or queuing commands. However w/ the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses w/o complexity of translation.
Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (w/ some faulting mechanism).
The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource XenGT also partitions them among VMs.
Quoted 3 lines
Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.
I think this looks good now overall, so please find someone from the i915 kernel team to review the details. You might need to poke managers to make that happen ;-)
-Daniel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 46 lines · Daniel Vetter
Quoted 45 lines · Yu Zhang
Intel GVT-g (previously known as XenGT), is a complete GPU virtualization solution with mediated pass-through for 4th generation Intel Core processors - Haswell platform. This technology presents a virtual full-fledged GPU to each Virtual Machine (VM). VMs can directly access performance-critical resources, without intervention from the hypervisor in most cases, while privileged operations from VMs are trap-and-emulated at minimal cost. For detail, please refer to
https://01.org/xen/blogs/wangbo85/2014/intel-gvt-gxengt-pubic-releaseThis patch set includes necessary code changes when i915 driver runs inside a VM. Though ideally we can run an unmodified i915 driver in VM, adding such enlightenments can greatly reduce the virtualization complexity in orders of magnitude. Code changes for the host side, which includes the actual Intel GVT-g implementation, will be sent out in another patchset.
The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory w/o hypervisor's intervention, e.g. filling textures or queuing commands. However w/ the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses w/o complexity of translation.
Note: The partitioning of global graphics memory may break some applications, with large objects in the aperture, because current userspace assumes half of the aperture usable. That would need separate fix either in user space (e.g. remove assumption in mesa) or in kernel (w/ some faulting mechanism).
The partitioning knowledge is conveyed through a reserved MMIO range, called PVINFO, which will be architecturally reserved in future hardware generations. Another information carried through PVINFO is about the number of fence registers. As a global resource XenGT also partitions them among VMs.
Quoted 8 lines
Quoted 3 lines
Other changes are trivial as optimizations, to either reduce the trap overhead or disable power management features which don't make sense in a virtualized environment.
I think this looks good now overall, so please find someone from the i915 kernel team to review the details. You might need to poke managers to make that happen ;-)
-Daniel
Thanks a lot, Daniel.
This is great. :-)
Yu
Quoted 1 line
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Hi,
I'll try to do the detailed review of your series in the following few days. I might ask some questions on the design also to help me understand the bigger picture.
First thing, I see that patches are checkpatch.pl clean, apart when run in strict mode. I think Daniel prefers "--strict" nowadays, at least I needed to fix up those in my patches so you should probably do the same.
On 11/13/2014 12:02 PM, Yu Zhang wrote:
[snip]
Quoted 7 lines
The primary change introduced here is to implement so-called "address space ballooning" technique. XenGT partitions global graphics memory among multiple VMs, so each VM can directly access a portion of the memory w/o hypervisor's intervention, e.g. filling textures or queuing commands. However w/ the partitioning an unmodified i915 driver would assume a smaller graphics memory starting from address ZERO, so requires XenGT
Quoted 8 lines
core module (vgt) to translate the graphics address between 'guest view' and 'host view', for all registers and command opcodes which contain a graphics memory address. To reduce the complexity, XenGT introduces "address space ballooning", by telling the exact partitioning knowledge to each guest i915 driver, which then reserves and prevents non-allocated portions from allocation. Then vgt module only needs to scan and validate graphics addresses w/o complexity of translation.
I couldn't figure this out - is there any hardware protection in there, or a virtual i915 instance could access memory outside it's area if there was a security bug/exploit in the driver, or the balloon was incorrectly set up?
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 10 lines · Tvrtko Ursulin
Hi,
I'll try to do the detailed review of your series in the following few days. I might ask some questions on the design also to help me understand the bigger picture.
First thing, I see that patches are checkpatch.pl clean, apart when run in strict mode. I think Daniel prefers "--strict" nowadays, at least I needed to fix up those in my patches so you should probably do the same.
Well I mostly run --strict since that checks for the alignment of continuation lines for function parameters. Imo not following the i915 coding style for that looks a bit too jarring ;-)
Still checkpatch is just a tool and occasionally seriously lacks taste, so please use --strict wisely instead of strictly.
-Daniel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Jani Nikula
maintainerQuoted 6 lines · Daniel Vetter
Well I mostly run --strict since that checks for the alignment of continuation lines for function parameters. Imo not following the i915 coding style for that looks a bit too jarring ;-)
Still checkpatch is just a tool and occasionally seriously lacks taste, so please use --strict wisely instead of strictly.
Perhaps we should use the --subjective option instead. Not that it would actually change anything, it's exactly the same as --strict. We'd still require the same things for i915, but we'd be openly subjective about it. ;)
BR,
Jani.
Signature · 5 lines
Jani Nikula, Intel Open Source Technology Center _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx