PAT and cache coherency support
Subsystems (4)
Re: [Intel-xe] [RFC 5/5] drm/xe/uapi: support pat_index selection with vm_bind
In reply to Matthew AuldQuoted 169 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Monday, September 4, 2023 5:24 PMQuoted 9 lines · Carl
Quoted 9 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Thursday, August 31, 2023 6:44 PMQuoted 2 lines · Carl
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Thursday, August 31, 2023 12:02 AMQuoted 2 lines · Carl
Several questions:
1. thepat_indexfromvm_bindwill override the setting from
bo_create?Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
How to keep the value from
bo_createunchanged?You only set the
pat_indexduringvm_bind. Atgem_createyou just need tell the kernel what CPU side caching mode to use for system memory (wb/wc/uc), plus the expected GPU coherency mode of thepat_index. The current expectation is that the coherency mode of thepat_indexshould match the one atgem_create.Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
It means that we set the incomplete
pat_indexatgem_create(coherency
only) And set another part atvm_bind(gpu cache) Why we separate them? Why not set a full value atvm_bind?You always give the full
pat_indexatvm_bind. It's platform specific but the BSpec for the most part tells you what the coherency mode of the
pat_indexis.Quoted 11 lines
Quoted 11 lines
For example if you look at the MTL
pat_indextable it tells you thecoh_mode:
pat_index0-2 =coh_mode"No snoop" =COH_NONEpat_index 3 = coh_mode "1way" = COH_1WAY pat_index 4 = coh_mode "2way" = COH_2WAYOn future platforms the table is larger and also encodes stuff like compression, but KMD really only cares about the
coh_mode, since that will restrict the allowed CPUsmem_cachingvalues that userspace canselect.
Quoted 3 lines
Quoted 3 lines
From KMD pov this matters since we always give userspace zeroed
memory, which is useful for normal applications but is also needed forsecurity reasons.
Quoted 10 lines
Quoted 10 lines
However the actual clearing might result in the writes only being visible in the CPU cache (not flushed to main memory), and if the GPU doesn't snoop the CPU cache it can read directly from main memory, which effectively bypasses the clearing. To prevent that the KMD needs to know how userspace will map the memory via the GPU i.e what is the
coh_modefor thepat_index. The gist is that if you selectCOH_NONEthen it needs to clflush the pages before giving them to userspace, which is done by forcing you to use uc/wc for thesmem_caching(the kernel will internally issue cflush on x86 when marking the pages as uc/wc). And if you select wb there is no flushing, buthere you need to use 1way or 2way.
Quoted 5 lines
Quoted 1 line
It sounds like a perf optimization, if no considering the perf, you
could always call Clflush to flush the cpu cache to main memory before giving it to user spaceI think the other thing is that the KMD is no longer doing any manual flushing, but instead the core kernel will now do that for us when marking the pages as wc/uc on x86.
Quoted 5 lines
Quoted 5 lines
AFAIK separating them is just to prevent userspace from mixing different coherency modes for the same object by having it immutable (you can still use different
pat_indexbut the coherency must becompatible).
Quoted 5 lines
Quoted 5 lines
Also at creation time we can determine if the
smem_cachingis compatible with thecoh_mode.Note that it might be that we ditch the
COH_2WAYand just haveCOH_NONEandCOH_AT_LEAST_1WAY, in which case this might be slightlydifferent.
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 4 lines
Quoted 4 lines
2. no UC/WB/WC definition (CPU cachable) in
drm_xe_gem_mmap_offset,Quoted 9 lines
will it be handled by KMD automatically?
Quoted 3 lines
For example: if set 1-way, it means GPU could snoop CPU cache, we could use WB in mmap offset If it is
COHERENCY_NONE, we could only useUC, all these logic is handled by KMD automatically?
Yes, the mmap will use the same CPU side caching mode that you set at
gem_createwithsmem_caching(wb/wc/uc).It means it could not be changed after
gem_create. Sometimes, UMD doesn’t know Whether the data need to be accessed by CPU atgem_create. only application Knows the purpose of allocation & the surface usage. So, may need change UMD api to let applications set whether theQuoted 7 lines
Quoted 7 lines
resource is "accessible", or it will hurt the perf.
Right, the
smem_cachingis immutable.Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
3. about " For imported dma-buf (from a different device) the coherency mode is also implicit and must also be either 1WAY or
2WAY"
Quoted 19 lines
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
it means it must be 1way or 2way, and UMD need not to set it?
If you are going to
vm_bindit you need to supply thepat_indexand in this case it needs to be either 1way or 2way. The object comes from a different device so we might not have smem_caching/coh_mode like we do for native objects.Quoted 7 lines
How about the GPU cache setting of imported dma-buf?
Is there possible conflict between dma-buf producer and consumer? For example:
In producer side , one bo is set GPU L3 cache, also CPU cacheable, it should be 2Way In client side, we know the coherency should be 1-way or 2-way, if it suppose no GPU cache Used. It just set 1-way. It may cause some problems?Quoted 3 lines
If the producer and consumer are the same device, then I think the original gem_create.coh_mode is considered, since it's effectively still a native object. In which case the normal rules apply.
Quoted 17 lines
But the consumer does not know the
coh_mode, so , it is possible that it set will set Another value, then it will failed.
And because consumer does not know it is 1-way or 2-way, consumer could not Set the gpu cache correctly. Because if it is 1-way, cpu could not snoop gpu cacheQuoted 11 lines
If it comes from a different device then we don't know what gem_create.coh_mode is (or if that even exists) since we don't have the usual native object, but just some opaque dma-buf object. In this case it must be at least 1way or 2way.
From KMD pov we only care about userspace not being able to bypass
the clearing. There might still be other coherency issues though.Quoted 2 lines
Also , when a bo is created, UMD does not know whether it will be
exported.
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
It maybe be set as
COHERENCY_NONE, right?Yeah, you can use whatever you want. Just that your
smem_cachingwill need to be wc/uc if usingCOH_NONE. On the importer side, if it comes from the same device then it's just a normal native object and the normalrules apply.
Quoted 4 lines
Quoted 4 lines
Currently this would mean also using
pat_indexwithCOH_NONE. If it is imported from a different device then you need to use apat_indexwith 1way or 2way, since KMD doesn't really know how the object is mapped on the CPU or if the CPU caches are dirty (potentially withclearing).
Quoted 7 lines
Quoted 3 lines
Is this going to be problematic?
If imported surfaces is from same device, how consumer know it is original
coh_nonOr 1-way or 2-way?And it also means producer and consumer could have different
pat_indexfor
Quoted 9 lines
same bo.
Ok, if KMD instead allows 1way or 2way for dma-buf when it's external or from the same device? Would that help here? i.e you can always select 1way or 2way, even if it came from the same device and was
coh_none?Other option is you could call the
vm_bindioctl to check if thecoh_modeis supported by checking if it returns an error? But maybe that is too nasy.
Still a bit confuse, I create a bo with coh_none, then export it to a PRIME fd. In consumer side, it does not know it is coh_none, and will call vm_bind to set pat_index.
You mean it will return a error when I set different coh value though vm_bind? And try coh_non, 1-way, 2-way, then get the correct one? TBH, it is ugly.
My concern is: if it is coh_non or 1-way, it means I could not use GPU cache if the bo also is CPU accessible. Because CPU could not snoop GPU cache.
So, GPU cache specified by pat_index also will be rejected?
Could we add some query interface to query the CPU cache and coherency setting of a bo?
Quoted 1,007 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 9 lines
Quoted 9 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Wednesday, August 30, 2023 7:28 PM
To: Roper, Matthew D <[email protected]>
Cc: [email protected]; Mishra, PallaviQuoted 3 lines
Quoted 3 lines
Thomas Hellström <[email protected]>; Joonas Lahtinen <[email protected]>; De Marchi, Lucas <[email protected]>; Souza, Jose <[email protected]>;
Hazubski,
Quoted 2 lines
Quoted 2 lines
Filip <[email protected]>; Zhang, Carl
<[email protected]>; Yu,Effie
Quoted 3 lines
Quoted 3 lines
<[email protected]>
Subject: Re: [RFC 5/5] drm/xe/uapi: supportpat_indexselection withQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
vm_bindQuoted 6 lines
Quoted 6 lines · Matt Roper
Quoted 4 lines · Matthew Auld
Quoted 3 lines
Allow userspace to directly control the
pat_indexfor a given vm binding. This should allow directly controlling the coherency, caching and potentially other stuff in the future for theppGTT binding.
Quoted 37 lines
Quoted 37 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
The exact meaning behind the
pat_indexis very platform specific (see BSpec or PRMs) but effectively maps to some predefined memory attributes. From the KMD pov we only care about the coherency that is provided by thepat_index, which falls into eitherNONE, 1WAY or 2WAY.
Quoted 30 lines
Quoted 30 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The
vm_bindcoherency mode for the givenpat_indexneeds to matchthe
Quoted 27 lines
Quoted 27 lines
Quoted 10 lines
Quoted 2 lines
given
coh_modethat was set at object creation. For platforms that lackIs it actually important to match the coherency mode? I think one of the main goals was to know up front if userspace might be using a non-snooping PAT setting that would let it bypass the CPU cache (and potentially read old, stale data from a different process if the buffer's clear value is still sitting in cache and hasn't landed in memory yet).
Quoted 6 lines
If that's the only concern, then I think it should still be fine to map with a non-matching PAT as long as it's more coherent than the one specified at creation, right? E.g., if the buffer was created with 1-way coherency, it would be fine to map it with 2-way because userspace still can't use that to observe the previous contents of the buffer. Or
Yeah, I guess we could in theory do something that.
Quoted 7 lines
if the buffer was created with "non-coherent" then we've already done the necessary clflushing in kernel before handing to buffer over to userspace to ensure the clear value landed in memory, so any valid PAT index should be safe (from a security POV) after that, right? Any other problems that arise from mismatched coherency would just be contained to the app possibly shooting itself in the foot, which isn't really our concern.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
That is also my understanding, at least from the KMD security pov. If you allocate as wb then you must use at least 1way, since there is no flushing
for
Quoted 1 line
Quoted 1 line
clearing or swap-in. For uc/wc you could in theory use whatever you
want.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
the explicit coherency mode, we treat UC/WT/WC as NONE and
WB
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
as
2WAY.
Quoted 2 lines
Quoted 2 lines
For userptr mappings we lack a corresponding gem object, so the
expected
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
coherency mode is instead implicit and must fall into either 1WAY or 2WAY. Trying to use NONE will be rejected by the kernel. For imported dma-buf (from a different device) the coherency mode is also implicit and must also be either 1WAY or
2WAY.
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
As part of adding
pat_indexsupport withvm_bindwe also need stopusing
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
xe_cache_leveland instead use thepat_indexin various places. We still make use ofxe_cache_level, but only as a convenience for kernel internal objectsi (internally it maps to some reasonablepat_index). ForQuoted 12 lines
Quoted 3 lines
Quoted 3 lines
Maybe we should kill
xe_cache_levelcompletely and just assignxe_gt->pat_cached/xe_gt->pat_uncachedat init that can be used in appropriate places, similar to what we do with MOCS (gt-mocs.uc_index,
Quoted 8 lines
Quoted 1 line
gt->mocs.wb_index)?OK, seems reasonable to me.
Quoted 4 lines
Quoted 4 lines
now this is just a 1:1 conversion of the existing code, however for platforms like MTL+ we might need to give more control through
bo_createQuoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
or stop using WB on the CPU side if we need CPU access.
Bspec: 45101, 44235 #xe
Bspec: 70552, 71582, 59400 #xe2Signed-off-by Matthew Auld <[email protected]>Cc Pallavi Mishra <[email protected]>Cc Thomas Hellström <[email protected]>Cc Joonas Lahtinen <[email protected]>Cc Lucas De Marchi <[email protected]>Cc Matt Roper <[email protected]>Cc José Roberto de Souza <[email protected]>Cc Filip Hazubski <[email protected]>Cc Carl Zhang <[email protected]>Cc Effie Yu <[email protected]>---
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/i915/gt/intel_gtt.h | 2 +- drivers/gpu/drm/xe/tests/xe_migrate.c | 2 +- drivers/gpu/drm/xe/xe_ggtt.c | 7 ++- drivers/gpu/drm/xe/xe_ggtt_types.h | 2 +- drivers/gpu/drm/xe/xe_migrate.c | 14 ++--- drivers/gpu/drm/xe/xe_pt.c | 32 +++++------- drivers/gpu/drm/xe/xe_pt.h | 6 +-- drivers/gpu/drm/xe/xe_vm.c | 73+++++++++++++++++++++----
Quoted 6 lines
Quoted 6 lines
-
Quoted 5 lines
Quoted 5 lines
-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm_types.h | 13 +++-- include/uapi/drm/xe_drm.h | 41 ++++++++++++++- 10 files changed, 134 insertions(+), 58 deletions(-)Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Review
Quoted 1 line
(quoted diff) +0
-0
b/drivers/gpu/drm/i915/gt/intel_gtt.hQuoted 5 lines
drivers/gpu/drm/i915/gt/intel_gtt.h struct i915_address_space { · line 302
Quoted 5 lines
index
4d6296cdbcfd..bb4c182048c3100644drivers/gpu/drm/i915/gt/intel_gtt.h +0
-0@@ -302,7 +302,7 @@ struct i915_address_space { (*alloc_scratch_dma)(struct i915_address_space*vm,
Quoted 33 lines
Quoted 33 lines
int
Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Review
Quoted 9 lines
drivers/gpu/drm/i915/gt/intel_gtt.h +1
-1sz);u64 (*pte_encode)(dma_addr_t addr, -unsigned int pat_index, +u32 pat_index,u32 flags);/* Create a valid PTE */ #define PTE_READ_ONLY BIT(0)#define PTE_LM BIT(1)(quoted diff) +0
-0
b/drivers/gpu/drm/xe/tests/xe_migrate.cQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/tests/xe_migrate.c static void xe_migrate_sanity_test(struct
Quoted 4 lines
index
5c8d5e78d9bc..7a128fd20a29100644drivers/gpu/drm/xe/tests/xe_migrate.c +0
-0@@ -301,7 +301,7 @@ static void xe_migrate_sanity_test(struct
xe_migrateQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
*m, struct kunit *test)
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/tests/xe_migrate.c +0
-0/* First part of the test, are we updating our pagetable bo with a new
entry? */
Quoted 1 line
Quoted 1 line
xe_map_wr(xe, &bo->vmap, XE_PAGE_SIZE *(
NUM_KERNEL_PDE-Quoted 11 lines
Quoted 11 lines
1),
Quoted 9 lines
Quoted 9 lines
u64,
Quoted 3 lines
drivers/gpu/drm/xe/tests/xe_migrate.c
Quoted 3 lines
0xdeaddeadbeefbeef);drivers/gpu/drm/xe/tests/xe_migrate.c +1
-1- expected =xe_pte_encode (pt,0 ,XE_CACHE_WB ,0 ); + expected = xe_pte_encode(pt, 0, xe_pat_get_index(xe,
XE_CACHE_WB), 0);Quoted 4 lines
Review
Quoted 4 lines
if (m->q->vm->flags & XE_VM_FLAG_64K) expected |= XE_PTE_PS64; if (xe_bo_is_vram(pt))(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_ggtt.cQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c u64 xe_ggtt_pte_encode(struct xe_bo *bo,
Quoted 4 lines
index
209fa053d9fb..4134c26150a5100644drivers/gpu/drm/xe/xe_ggtt.c +0
-0@@ -41,7 +41,8 @@ u64 xe_ggtt_pte_encode(struct xe_bo *bo,u64
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
bo_offset)Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c u64 xe_ggtt_pte_encode(struct xe_bo *bo, · line 44
Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c +0
-1pte |=XE_GGTT_PTE_DM ;if ((ggtt->pat_encode).pte_encode) - pte = (ggtt->pat_encode).pte_encode(xe, pte,
XE_CACHE_WB_1_WAY);Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_ggtt.c u64 xe_ggtt_pte_encode(struct xe_bo *bo, · line 45
Quoted 2 lines
drivers/gpu/drm/xe/xe_ggtt.c +2
-0+ pte = (ggtt->pat_encode).pte_encode(xe, pte, +xe_pat_get_index(xe,
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
XE_CACHE_WB_1_WAY));Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c static void primelockdep(struct xe_ggtt
Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c +0
-0return pte; }@@ -102,10 +103,8 @@ static void primelockdep(struct xe_ggtt*ggtt)
Quoted 86 lines
Quoted 86 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c +0
-0} static u64 xelpg_ggtt_pte_encode_pat(struct xe_device *xe,u64
pte_pat,Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_ggtt.c
Quoted 1 line
drivers/gpu/drm/xe/xe_ggtt.c +0
-1-enum xe_cache_level cache)
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Review
Quoted 8 lines
drivers/gpu/drm/xe/xe_ggtt.c +1
-2+ u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELPG_GGTT_PTE_PAT_MASK );if (pat_index &BIT (0 ))(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_ggtt_types.hQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_ggtt_types.h struct xe_ggtt { · line 34
Quoted 8 lines
index
7e55fac1a8a9..0bc40cb072e3100644drivers/gpu/drm/xe/xe_ggtt_types.h +0
-1@@ -31,7 +31,7 @@ struct xe_ggtt {struct { u64 (*pte_encode)(struct xe_device *xe, u64 pte_pat, - enum xe_cache_levelcache);
Quoted 32 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Review
Quoted 5 lines
drivers/gpu/drm/xe/xe_ggtt_types.h +1
-0+ u32 pat_index); } pat_encode; };(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_migrate.cQuoted 16 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
index
a782ea282cb6..54585e98452a100644drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -24,6 +24,7 @@#include "xe_lrc.h" Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_migrate.c +1
-0#include "xe_map.h" #include "xe_mocs.h" +#include "xe_pat.h" #include "xe_pt.h" #include "xe_res_cursor.h" #include "xe_sched_job.h" @@ -162,6 +163,7 @@ static int xe_migrate_prepare_vm(struct
xe_tile*tile,
Quoted 2 lines
Quoted 2 lines
struct
xe_migrate*m,Quoted 1 line
Quoted 1 line
u32
num_entries=NUM_PT_SLOTS,num_level= vm-pt_root[id]-
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
level;
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct
Quoted 7 lines
u32 map_ofs, level, i; struct xe_bo *bo, *batch = tile->mem.kernel_bb_pool->bo;drivers/gpu/drm/xe/xe_migrate.c +1
-0+u32 pat_index =xe_pat_get_index (xe,XE_CACHE_WB );u64 entry;int ret;@@ -189,14 +191,14 @@ static int xe_migrate_prepare_vm(struct
xe_tileQuoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
*tile, struct
xe_migrate*m,Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 192
Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-1return ret; } - entry = xe_pde_encode(bo, bo->size - XE_PAGE_SIZE,
XE_CACHE_WB);Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 194
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ entry = xe_pde_encode(bo, bo->size - XE_PAGE_SIZE,
pat_index);Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 199
Quoted 7 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-1xe_pt_write (xe, &vm->pt_root[id]->bo->vmap,0 , entry); map_ofs = (num_entries - num_level) *XE_PAGE_SIZE ;/* Map the entire BO in our level 0 pt */ for (i =0 , level =0 ; i < num_entries; level++) { - entry = xe_pte_encode(bo, i * XE_PAGE_SIZE,
XE_CACHE_WB,Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
0);
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 201
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ entry = xe_pte_encode(bo, i * XE_PAGE_SIZE,
pat_index, 0);Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 202
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-0xe_map_wr(xe, &bo->vmap, map_ofs + level * 8, u64,entry);
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -214,7 +216,7 @@ static int xe_migrate_prepare_vm(struct
xe_tile*tile,
Quoted 3 lines
Quoted 3 lines
struct
xe_migrate*m,Quoted 2 lines
Quoted 2 lines
for (i = 0; i < batch->size; i += vm->flags & XE_VM_FLAG_64K ?Quoted 7 lines
Quoted 7 lines
XE_64K_PAGE_SIZE:Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 2 lines
XE_PAGE_SIZE) {drivers/gpu/drm/xe/xe_migrate.c +0
-1- entry = xe_pte_encode(batch, i,
XE_CACHE_WB, 0);Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ entry =xe_pte_encode (batch, i, pat_index,0 ); xe_map_wr(xe, &bo->vmap, map_ofs + level*
Quoted 5 lines
Quoted 5 lines
8, u64,
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
entry);drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -239,7 +241,7 @@ static int xe_migrate_prepare_vm(struct
xe_tile*tile,
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
struct
xe_migrate*m,Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 4 lines
flags =
XE_PDE_64K;entry = xe_pde_encode(bo,
map_ofs+ (level - 1) *drivers/gpu/drm/xe/xe_migrate.c +0
-1- XE_PAGE_SIZE,
XE_CACHE_WB);Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ XE_PAGE_SIZE, pat_index); xe_map_wr(xe, &bo->vmap, map_ofs +
XE_PAGE_SIZEQuoted 7 lines
Quoted 7 lines
* level,
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
u64,
Quoted 4 lines
Quoted 4 lines
entry | flags); }drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -247,7 +249,7 @@ static int xe_migrate_prepare_vm(struct
xe_tile*tile,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
struct
xe_migrate*m,Quoted 7 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 7 lines
/* Write PDE's that point to our BO. */ for (i = 0; i < num_entries - num_level; i++) { entry = xe_pde_encode(bo, i * XE_PAGE_SIZE,drivers/gpu/drm/xe/xe_migrate.c +1
-1- XE_CACHE_WB); + pat_index); xe_map_wr(xe, &bo->vmap, map_ofs +
XE_PAGE_SIZEQuoted 130 lines
Quoted 130 lines
+
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(i + 1) * 8, u64, entry); @@ -1256,7 +1258,7@@
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
xe_migrate_update_pgtables(struct
xe_migrateQuoted 5 lines
Quoted 5 lines
*m,
Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 4 lines
XE_WARN_ON(pt_bo->size != SZ_4K);drivers/gpu/drm/xe/xe_migrate.c +0
-1- addr = xe_pte_encode(pt_bo, 0,
XE_CACHE_WB, 0);Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ addr = xe_pte_encode(pt_bo, 0,xe_pat_get_index(xe,
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
XE_CACHE_WB), 0);Quoted 10 lines
Quoted 10 lines
bb->cs[bb->len++] = lower_32_bits(addr); bb->cs[bb->len++] = upper_32_bits(addr); }(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_pt.cindex64713f400d94..019af2920078100644drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -10,6 +10,7 @@#include "xe_gt.h" Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_pt.c +1
-0#include "xe_gt_tlb_invalidation.h" #include "xe_migrate.h" +#include "xe_pat.h" #include "xe_pt_types.h" #include "xe_pt_walk.h" #include "xe_res_cursor.h" @@ -57,24 +58,22 @@ static struct xe_pt *xe_pt_entry(struct
xe_pt_dir
*pt_dir, unsigned int index)Quoted 12 lines
Quoted 12 lines
* * Return: An encoded page directory entry. No errors. */drivers/gpu/drm/xe/xe_pt.c +2
-2-u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level cache) +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 +pat_index) {u64 pde;struct xe_vm *vm = bo->vm;struct xe_device *xe = vm->xe;Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c +1
-2- pde =xe_bo_addr (bo, bo_offset,XE_PAGE_SIZE ); pde |=XE_PAGE_PRESENT |XE_PAGE_RW ;if ((vm->pat_encode).pde_encode) - pde = (vm->pat_encode).pde_encode (xe, pde, cache); + pde = (vm->pat_encode).pde_encode(xe, pde,
pat_index);Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 9 lines
return pde; }drivers/gpu/drm/xe/xe_pt.c +1
-1-static u64 __pte_encode (u64 pte,enum xe_cache_level cache, +static u64 __pte_encode (u64 pte,u32 pat_index,struct xe_vma *vma,u32 pt_level) { struct xe_vm *vm = xe_vma_vm(vma); @@ -89,7 +88,7 @@static
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
u64 __pte_encode(u64 pte, enum
xe_cache_levelcache,Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 5 lines
pte |= XE_PTE_NULL; if ((vm->pat_encode).pte_encode)drivers/gpu/drm/xe/xe_pt.c +1
-1- pte = (vm->pat_encode).pte_encode (xe, pte, cache); + pte = (vm->pat_encode).pte_encode(xe, pte,
pat_index);Quoted 47 lines
Quoted 21 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c static u64 __pte_encode(u64 pte, enum
Quoted 4 lines
if (pt_level == 1) pte |= XE_PDE_PS_2M;drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -112,7 +111,7 @@ static u64 __pte_encode(u64 pte, enum
xe_cache_levelcache,Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c +0
-1* * Return: An encoded page-table entry. No errors. */ -u64 xe_pte_encode(struct xe_bo *bo, u64 offset, enum
xe_cache_levelQuoted 6 lines
Quoted 6 lines
cache,
Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c u64 xe_pte_encode(struct xe_bo *bo, u64
Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c +1
-0+u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 pat_index,u32 pt_level) {u64 pte;@@ -121,7 +120,7 @@ u64 xe_pte_encode(struct xe_bo *bo, u64offset,
Quoted 4 lines
Quoted 4 lines
enum
xe_cache_levelcache,Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c +0
-0if (xe_bo_is_vram (bo) ||xe_bo_is_stolen_devmem (bo)) pte |=XE_PPGTT_PTE_DM ;Quoted 19 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c +1
-1-return __pte_encode (pte, cache,NULL , pt_level); +return __pte_encode (pte, pat_index,NULL , pt_level); } static u64 __xe_pt_empty_pte(struct xe_tile *tile, struct
xe_vm*vm, @@ -134,12 +133,12 @@ static u64
__xe_pt_empty_pte(structxe_tile*tile,
Quoted 11 lines
Quoted 11 lines
struct
xe_vm*vm,Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 5 lines
if (level == 0) { u64 empty = xe_pte_encode(vm->scratch_bo[id], 0,drivers/gpu/drm/xe/xe_pt.c +1
-1- XE_CACHE_WB, 0); + xe_pat_get_index(vm->xe,
XE_CACHE_WB), 0);Quoted 4 lines
Quoted 4 lines
return empty; } else { return xe_pde_encode(vm->scratch_pt[id][level - 1]-bo, 0,
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 2 lines
drivers/gpu/drm/xe/xe_pt.c +1
-1- XE_CACHE_WB); + xe_pat_get_index(vm->xe,
XE_CACHE_WB));Quoted 3 lines
Quoted 3 lines
} }Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c struct xe_pt_stage_bind_walk { · line 370
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c +0
-2@@ -368,8 +367,6 @@ struct xe_pt_stage_bind_walk {struct xe_vm *vm;/** @tile: The tile we're building for. */ struct xe_tile *tile; -/** @cache: Desired cache level for the ptes */ -enum xe_cache_level cache;/** @default_pte: PTE flag only template. No address is associated */
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +1
-1u64 default_pte; /** @dma_offset: DMA offset to add to the PTE. */ @@ -604 ,7 +601,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent,Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
pgoff_toffset,Quoted 3 lines
Quoted 3 lines
pte = __pte_encode(is_null ? 0 : xe_res_dma(curs) + xe_walk-
dma_offset,Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +0
-1- xe_walk->cache, xe_walk->vma,level);
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +1
-0+ xe_walk->vma->pat_index,xe_walk-
Quoted 64 lines
Quoted 4 lines
Quoted 4 lines
vma,
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
level);
Quoted 2 lines
Quoted 2 lines
pte |= xe_walk->default_pte;Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind_entry(struct xe_ptw · line 666
Quoted 3 lines
/*drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -669,7 +666,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent,
pgoff_toffset,Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind_entry(struct xe_ptw · line 673
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +0
-1xe_child->is_compact = true; } - pte = xe_pde_encode(xe_child->bo, 0, xe_walk-cache)
Quoted 51 lines
|
Quoted 4 lines
Quoted 4 lines
flags;
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind_entry(struct xe_ptw · line 670
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +1
-0+ pte = xe_pde_encode(xe_child->bo, 0, xe_walk->vma-
pat_index) | flags;Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +0
-0ret = xe_pt_insert_entry(xe_walk, xe_parent, offset,
xe_child,Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +0
-0pte);}drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -730,13 +727,8 @@ xe_pt_stage_bind(struct xe_tile *tile,struct
xe_vmaQuoted 2 lines
Quoted 2 lines
*vma,
Quoted 1 line
Quoted 1 line
if (vma && vma->gpuva.flags &
XE_VMA_ATOMIC_PTE_BIT)Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
xe_walk.default_pte |=
XE_USM_PPGTT_PTE_AE;Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 5 lines
Quoted 1 line
xe_walk.dma_offset = vram_region_gpu_offset(bo-
ttm.resource);
drivers/gpu/drm/xe/xe_pt.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c +0
-3- xe_walk.cache =XE_CACHE_WB ; - } else { - if (!xe_vma_has_no_bo(vma) && bo->flags &
XE_BO_SCANOUT_BIT)Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c +1
-3- xe_walk.cache =XE_CACHE_WT ; - else - xe_walk.cache = XE_CACHE_WB; } +if (!xe_vma_has_no_bo (vma) &&xe_bo_is_stolen (bo)) xe_walk.dma_offset =xe_ttm_stolen_gpu_offset(xe_bo_device(bo));
Quoted 8 lines
Quoted 8 lines
(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_pt.hindex01be7ab08f87..1d433a5a96b4100644drivers/gpu/drm/xe/xe_pt.h +0
-0@@ -45,10 +45,8 @@ __xe_pt_unbind_vma(struct xe_tile *tile,struct
xe_vma*vma, structxe_exec_queuQuoted 11 lines
Quoted 11 lines
bool xe_pt_zap_ptes(struct
xe_tile*tile, structxe_vma
*vma);drivers/gpu/drm/xe/xe_pt.h +2
-3-u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level level); +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 +pat_index); -u64 xe_pte_encode(struct xe_bo *bo, u64 offset, enum
xe_cache_levelcache,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.h
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.h +2
-1- u32 pt_level); +u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 pat_index, +u32
pt_level);Quoted 4 lines
Quoted 4 lines
#endif(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_vm.cindex
7eeeed0411f3..34603a7e84b0Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
100644
drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -6,6 +6,7 @@#include "xe_vm.h" Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +1
-0#include <linux/dma-fence-array.h> +#include <linux/nospec.h> #include <drm/drm_print.h> #include <drm/ttm/ttm_execbuf_util.h> @@ -874,7 +875,8 @@static struct
xe_vma*xe_vma_create(struct
xe_vmQuoted 1 line
Quoted 1 line
*vm,
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c static struct xe_vma
Quoted 10 lines
u64 start, u64 end, bool read_only, bool is_null,drivers/gpu/drm/xe/xe_vm.c +2
-1- u8 tile_mask) + u8 tile_mask, + u32 pat_index) {struct xe_vma *vma;struct xe_tile *tile;@@ -913,6 +915,8 @@ static struct xe_vma*xe_vma_create(struct
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
xe_vmQuoted 10 lines
Quoted 10 lines
*vm,
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c static void
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c +2
-0vma->tile_mask |=0x1 << id; } + vma->pat_index = pat_index; +if (vm->xe->info.platform ==XE_PVC ) vma->gpuva.flags |=XE_VMA_ATOMIC_PTE_BIT ;@@ -1194,10 +1198,8 @@ static voidxe_vma_op_work_func(struct
Quoted 25 lines
Quoted 25 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
work_struct*w);Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0static void vm_destroy_work_func (struct work_struct *w); static u64 xelp_ppgtt_pde_encode_pat(struct xe_device *xe,u64
pde_pat,Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-1-enum xe_cache_level cache)
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c static u64
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c +1
-2+ u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pde_pat &= ~(XELP_PDE_PAT_MASK );if (pat_index &BIT (0 ))@@ -1213,10 +1215,8 @@ static u64xelp_ppgtt_pde_encode_pat(struct
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
xe_device*xe, u64pde_pat,Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0} static u64 xelp_ppgtt_pte_encode_pat(struct xe_device *xe,u64
pte_pat,Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-1-enum xe_cache_level cache)
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c struct xe_vm *xe_vm_lookup(struct
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c +1
-2+ u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELP_PTE_PAT_MASK );if (pat_index &BIT (0 ))@@ -1622,7 +1622,7 @@ struct xe_vm *xe_vm_lookup(struct
xe_fileQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
*xef,
Quoted 2 lines
Quoted 2 lines
u32 id)
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0u64 xe_vm_pdp4_descriptor(struct xe_vm *vm, struct xe_tile*tile)
Quoted 71 lines
Quoted 71 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c struct xe_vm *xe_vm_lookup(struct · line 1625
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1{ return xe_pde_encode(vm->pt_root[tile->id]->bo, 0, -XE_CACHE_WB ); + xe_pat_get_index(vm->xe,
XE_CACHE_WB));Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c static void print_op(struct xe_device · line 2311
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0} static struct dma_fence *@@ -2311,7 +2311,7 @@ static void print_op(struct xe_device *xe,struct
Quoted 4 lines
Quoted 4 lines
drm_gpuva_op*op)Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0static struct drm_gpuva_ops *vm_bind_ioctl_ops_create (struct xe_vm *vm,struct xe_bo *bo, u64 bo_offset_or_userptr, u64 addr, u64range,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c static void print_op(struct xe_device · line 2315
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- u32 operation, u8 tile_mask, u32 region) + u32 operation, u8 tile_mask, u32 region, u32
pat_index)Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0{struct drm_gem_object *obj = bo ? &bo->ttm.base :NULL ;struct ww_acquire_ctx ww; @@ -2339,6 +2339,7 @@vm_bind_ioctl_ops_create(struct
xe_vm*vm,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
struct
xe_bo*bo,Quoted 1 line
Quoted 1 line
struct
xe_vma_op*op =
gpuva_op_to_vma_op(__op);Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 5 lines
op->tile_mask = tile_mask;drivers/gpu/drm/xe/xe_vm.c +1
-0+ op->pat_index = pat_index; op->map.immediate = operation &
XE_VM_BIND_FLAG_IMMEDIATE;Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c vm_bind_ioctl_ops_create(struct xe_vm
Quoted 2 lines
op->map.read_only =drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2366,6 +2367,7 @@ vm_bind_ioctl_ops_create(struct xe_vm*vm,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
struct
xe_bo*bo,Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0struct xe_vma_op *op =
gpuva_op_to_vma_op(__op);Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c vm_bind_ioctl_ops_create(struct xe_vm
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c +1
-0op->tile_mask = tile_mask; + op->pat_index = pat_index; op->prefetch.region = region; }break ;@@ -2408,7 +2410,8 @@ vm_bind_ioctl_ops_create(struct xe_vm*vm,
Quoted 24 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
struct
xe_bo*bo,Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c vm_bind_ioctl_ops_create(struct xe_vm · line 2412
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0} static struct xe_vma *new_vma(struct xe_vm *vm, struct
drm_gpuva_op_map*op,Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c vm_bind_ioctl_ops_create(struct xe_vm · line 2416
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c +2
-1- u8 tile_mask, bool read_only, bool is_null) + u8 tile_mask, bool read_only, bool is_null, + u32 pat_index) { struct xe_bo *bo = op->gem.obj ? gem_to_xe_bo(op-gem.obj) :
Quoted 13 lines
NULL;
Quoted 12 lines
Quoted 12 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0struct xe_vma *vma;@@ -2425,7 +2428,7 @@ static struct xe_vma *new_vma(struct
xe_vm*vm, struct
drm_gpuva_op_map*op,Quoted 8 lines
Quoted 8 lines
vma = xe_vma_create(vm, bo, op->gem.offset, op->va.addr, op->va.addr + op->va.range - 1, read_only, is_null,drivers/gpu/drm/xe/xe_vm.c +1
-1- tile_mask); + tile_mask, pat_index);if (bo)xe_bo_unlock (bo, &ww);Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c static int
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2539,7 +2542,7 @@ static intvm_bind_ioctl_ops_parse(struct
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
xe_vmQuoted 3 lines
Quoted 3 lines
*vm, struct
xe_exec_queue*q,Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0vma = new_vma(vm, &op-base.map,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0op->tile_mask, op-Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
map.read_only,
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2544
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- op->map.is_null); + op->map.is_null, op-
pat_index);Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c static int
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2567,7 +2570,7 @@ static intvm_bind_ioctl_ops_parse(struct
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
xe_vmQuoted 5 lines
Quoted 5 lines
*vm, struct
xe_exec_queue*q,Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2570
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0vma = new_vma(vm, op-base.remap.prev,
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0op->tile_mask,Quoted 8 lines
Quoted 8 lines
read_only,Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2572
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- is_null); + is_null, op-
pat_index);Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c static int
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2603,7 +2606,7 @@ static intvm_bind_ioctl_ops_parse(struct
Quoted 22 lines
Quoted 22 lines
Quoted 6 lines
Quoted 6 lines
xe_vmQuoted 5 lines
Quoted 5 lines
*vm, struct
xe_exec_queue*q,Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2606
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0vma = new_vma(vm, op-base.remap.next,
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0op->tile_mask,
read_only,Quoted 15 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2608
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- is_null); + is_null, op-
pat_index);Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -3158,8 +3161,14 @@ static intvm_bind_ioctl_check_args(struct
xe_device*xe,Quoted 6 lines
Quoted 6 lines
u32 obj = (*bind_ops)[i].obj; u64 obj_offset = (*bind_ops)[i].obj_offset; u32 region = (*bind_ops)[i].region;drivers/gpu/drm/xe/xe_vm.c +1
-0+u32 pat_index = (*bind_ops)[i].pat_index;bool is_null = op &XE_VM_BIND_FLAG_NULL ;Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+ if (XE_IOCTL_DBG(xe, pat_index >= xe-info.pat_table_n_entries)) {
drivers/gpu/drm/xe/xe_vm.c
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +4
-0+ err = -EINVAL ; + goto free_bind_ops; + } +if (i ==0 ) { *async = !!(op &
XE_VM_BIND_FLAG_ASYNC);Quoted 40 lines
Quoted 40 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
} else if (XE_IOCTL_DBG(xe, !*async) || @@ -3346,8+3355,25
Quoted 15 lines
Quoted 14 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
@@ int xe_vm_bind_ioctl(struct
drm_device*dev,
Quoted 12 lines
Quoted 12 lines
void *data, struct
drm_file*file)Quoted 11 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 11 lines
struct drm_gem_object *gem_obj; u64 range = bind_ops[i].range; u64 addr = bind_ops[i].addr;drivers/gpu/drm/xe/xe_vm.c +6
-0+u32 op = bind_ops[i].op;u32 obj = bind_ops[i].obj;u64 obj_offset = bind_ops[i].obj_offset; +u32 pat_index = bind_ops[i].pat_index; +u16 coh_mode; + + pat_index = array_index_nospec(pat_index, + xe-info.pat_table_n_entries);
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+ coh_mode = xe_pat_index_get_coh_mode(xe,
pat_index);Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+if (XE_IOCTL_DBG (xe, !coh_mode)) {Assuming we drop the unusable entries from the TGL table, this should be impossible, right? Any index that makes it past the
n_entriescheck at the top of the function should have a valid, non-zero
coh_modevalue.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
So this should probably be an assertion (to highlight a KMD bug) rather than just a silent uapi failure return.
Makes sense.
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c +5
-0+ err = -EINVAL ; + goto put_obj; + } + + if (XE_IOCTL_DBG(xe, VM_BIND_OP(op) ==
XE_VM_BIND_OP_MAP_USERPTR&&Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+ coh_mode ==
XE_GEM_COHERENCY_NONE))Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
{
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c +3
-0+ err = -EINVAL ; + goto put_obj; + }if (!obj)continue ;@@ -3375,6 +3401,22 @@ int xe_vm_bind_ioctl(struct
drm_deviceQuoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
*dev,
Quoted 13 lines
Quoted 13 lines
void *data, struct
drm_file*file)Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3406
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +3
-0goto put_obj; } } + +if (bos[i]->coh_mode) { + if (XE_IOCTL_DBG(xe, bos[i]->coh_mode !=
coh_mode)) {Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3410
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +4
-0+ err = -EINVAL ; + goto put_obj; + } + } else if (XE_IOCTL_DBG(xe, coh_mode ==
XE_GEM_COHERENCY_NONE)) {Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3412
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +2
-0+/* +* Imported dma-buf from a different device should
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3413
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+* require 1way or 2way coherency since we don't
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
know
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3414
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+* how it was mapped on CPU. Just assume is it
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c +5
-0+* potentially cached on CPU side. +*/ + err = -EINVAL ; + goto put_obj; + } }if (args->num_syncs) {@@ -3412,10 +3454,11 @@ int xe_vm_bind_ioctl(struct
drm_deviceQuoted 39 lines
Quoted 39 lines
Quoted 8 lines
Quoted 8 lines
*dev,
Quoted 7 lines
Quoted 7 lines
void *data, struct
drm_file*file)Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3459
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +1
-0u64 obj_offset = bind_ops[i].obj_offset;u8 tile_mask = bind_ops[i].tile_mask;u32 region = bind_ops[i].region; +u32 pat_index = bind_ops[i].pat_index; ops[i] = vm_bind_ioctl_ops_create(vm, bos[i],
obj_offset,Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3460
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0addr, range, op,
tile_mask,Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
Review
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- region); + region, pat_index);if (IS_ERR (ops[i])) { err =PTR_ERR (ops[i]); ops[i] =NULL ;(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_vm_types.hQuoted 7 lines
Quoted 7 lines
index
83a1f87b6537..508679ed3c74100644drivers/gpu/drm/xe/xe_vm_types.h +0
-0@@ -111,6 +111,11 @@ struct xe_vma {*/ u8 tile_present;Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vma { · line 115
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm_types.h +2
-0+/** +* @pat_index: The pat index to use when encoding the PTEs for
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vma { · line 116
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +1
-0+thisvma.
Quoted 9 lines
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm_types.h +3
-0+*/ +u32 pat_index; +struct {struct list_head rebind_link; } notifier;@@ -338,10 +343,8 @@ struct xe_vm {bool batch_invalidate_tlb;Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 342
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm_types.h +0
-2struct { -u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat, - enum xe_cache_levelcache);
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 343
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +0
-1- u64 (*pde_encode)(struct xe_device *xe, u64
pde_pat,Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 344
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +0
-1-enum xe_cache_level cache);
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 346
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +1
-0+ u64 (*pte_encode)(struct xe_device *xe, u64 pte_pat,u32
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
pat_index);Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 347
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +1
-0+ u64 (*pde_encode)(struct xe_device *xe, u64
pde_pat,Quoted 40 lines
Quoted 40 lines
u32
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
pat_index);Quoted 21 lines
Quoted 21 lines
drivers/gpu/drm/xe/xe_vm_types.h +2
-0} pat_encode; };@@ -417,6 +420,8 @@ struct xe_vma_op {struct async_op_fence *fence;/** @tile_mask: gt mask for this operation */ u8 tile_mask; +/** @pat_index: The pat index to use for this operation. */ +u32 pat_index;/** @flags: operation flags */ enum xe_vma_op_flags flags;(quoted diff) +0
-0
b/include/uapi/drm/xe_drm.hindex64bc66d4b550..0c15b6f32447100644include/uapi/drm/xe_drm.h +0
-0@@ -600,8 +600,45 @@ struct drm_xe_vm_bind_op {*/ __u32 obj;Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 604
Quoted 4 lines
include/uapi/drm/xe_drm.h +2
-2-/** @pad: MBZ */ -__u32 pad; +/** +* @pat_index: The platform defined @pat_index to use for this
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
mapping.
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 605
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* The index basically maps to some predefined memory attributes,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 607
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * including things like caching, coherency and likely other +stuffin
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 609
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * the future. The exact meaning of the pat_index is +platform specificBTW, "other stuff in the future" already includes compression on Xe2, we just haven't landed the patches for the Xe2 table yet.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Ok, good to know.
Quoted 2 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 610
Quoted 2 lines
include/uapi/drm/xe_drm.h +1
-0+* and defined in the Bspec and PRMs. When the KMD sets up
Quoted 85 lines
Quoted 85 lines
the
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
binding
Quoted 4 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 614
Quoted 4 lines
include/uapi/drm/xe_drm.h +4
-0+* the index here is encoded into the ppGTT PTE. +* +* For coherency the @pat_index needs to match the +* drm_xe_gem_create.coh_mode, so either
XE_GEM_COHERENCY_NONE,Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 615
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* XE_GEM_COHERENCY_1WAY or
XE_GEM_COHERENCY_2WAY. TheQuoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
KMD will extract
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 616
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* the coherency mode from the @pat_index and reject if there is a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 619
Quoted 3 lines
include/uapi/drm/xe_drm.h +3
-0+* mismatch (see note below for pre-MTL platforms). +* +* Note: On pre-MTL platforms there is only a caching mode and no
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 620
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* explicit coherency mode, but on such hardware there is always a
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 622
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * shared-LLC (or is dgpu) so all GT memory accesses are +coherentwith
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 623
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* CPU caches even with the caching mode set as uncached. It's
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 624
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+onlythe
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 625
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* display engine that is incoherent (on dgpu it must be in VRAM which
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 626
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* is always mapped as WC on the CPU). However to keep the uapi
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
somewhat
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 627
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* consistent with newer platforms the KMD groups the different
Quoted 13 lines
Quoted 13 lines
cache
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 628
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* levels into the following coherency buckets on all pre-MTL platforms:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 633
Quoted 5 lines
include/uapi/drm/xe_drm.h +5
-0+* +* ppGTT UC -> XE_GEM_COHERENCY_NONE +* ppGTT WC -> XE_GEM_COHERENCY_NONE +* ppGTT WT -> XE_GEM_COHERENCY_NONE +* ppGTT WB -> XE_GEM_COHERENCY_2WAY As noted on the previous patch, it seems like 2-way is appropriate for LLC platforms, but 1-way might be a more accurate description of dGPU behavior.
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 636
Quoted 3 lines
include/uapi/drm/xe_drm.h +3
-0+* +* In practice UC/WC/WT should only ever used for scanout +surfaces on
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 637
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* such platforms since it is only the display engine that is actually
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 639
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * incoherent. Everything else should typically use WB given +that weWhat if we're sharing our buffers with some other (non-GPU) device? Are there cases where that other device wouldn't be coherent with the LLC, so we'd want to use one of these?
Yeah, I guess there might be cases like that. I'll reword.
Quoted 2 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Matt
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 640
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* have a shared-LLC. On MTL+ this completely changes (also potentially
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 641
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* no shared-LLC) and the HW defines the coherency mode as part
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 642
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+ofthe
Quoted 3 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op {
Quoted 3 lines
include/uapi/drm/xe_drm.h +3
-0+* @pat_index. +* +* Note: For userptr and externally imported dma-buf the kernel
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
expects
Quoted 10 lines
Quoted 9 lines
include/uapi/drm/xe_drm.h +3
-1+* either 1WAY or 2WAY for the @pat_index. +*/ +__u32 pat_index;union {/** -- 2.41.0
Re: [Intel-xe] [RFC 5/5] drm/xe/uapi: support pat_index selection with vm_bind
Quoted 179 lines · Carl
Quoted 169 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Monday, September 4, 2023 5:24 PMQuoted 9 lines · Carl
Quoted 9 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Thursday, August 31, 2023 6:44 PMQuoted 2 lines · Carl
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Thursday, August 31, 2023 12:02 AMQuoted 2 lines · Carl
Several questions:
1. thepat_indexfromvm_bindwill override the setting from
bo_create?Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
How to keep the value from
bo_createunchanged?You only set the
pat_indexduringvm_bind. Atgem_createyou just need tell the kernel what CPU side caching mode to use for system memory (wb/wc/uc), plus the expected GPU coherency mode of thepat_index. The current expectation is that the coherency mode of thepat_indexshould match the one atgem_create.Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
It means that we set the incomplete
pat_indexatgem_create(coherency
only) And set another part atvm_bind(gpu cache) Why we separate them? Why not set a full value atvm_bind?You always give the full
pat_indexatvm_bind. It's platform specific but the BSpec for the most part tells you what the coherency mode of the
pat_indexis.Quoted 11 lines
Quoted 11 lines
For example if you look at the MTL
pat_indextable it tells you thecoh_mode:
pat_index0-2 =coh_mode"No snoop" =COH_NONEpat_index 3 = coh_mode "1way" = COH_1WAY pat_index 4 = coh_mode "2way" = COH_2WAYOn future platforms the table is larger and also encodes stuff like compression, but KMD really only cares about the
coh_mode, since that will restrict the allowed CPUsmem_cachingvalues that userspace canselect.
Quoted 3 lines
Quoted 3 lines
From KMD pov this matters since we always give userspace zeroed
memory, which is useful for normal applications but is also needed forsecurity reasons.
Quoted 10 lines
Quoted 10 lines
However the actual clearing might result in the writes only being visible in the CPU cache (not flushed to main memory), and if the GPU doesn't snoop the CPU cache it can read directly from main memory, which effectively bypasses the clearing. To prevent that the KMD needs to know how userspace will map the memory via the GPU i.e what is the
coh_modefor thepat_index. The gist is that if you selectCOH_NONEthen it needs to clflush the pages before giving them to userspace, which is done by forcing you to use uc/wc for thesmem_caching(the kernel will internally issue cflush on x86 when marking the pages as uc/wc). And if you select wb there is no flushing, buthere you need to use 1way or 2way.
Quoted 5 lines
Quoted 1 line
It sounds like a perf optimization, if no considering the perf, you
could always call Clflush to flush the cpu cache to main memory before giving it to user spaceI think the other thing is that the KMD is no longer doing any manual flushing, but instead the core kernel will now do that for us when marking the pages as wc/uc on x86.
Quoted 5 lines
Quoted 5 lines
AFAIK separating them is just to prevent userspace from mixing different coherency modes for the same object by having it immutable (you can still use different
pat_indexbut the coherency must becompatible).
Quoted 5 lines
Quoted 5 lines
Also at creation time we can determine if the
smem_cachingis compatible with thecoh_mode.Note that it might be that we ditch the
COH_2WAYand just haveCOH_NONEandCOH_AT_LEAST_1WAY, in which case this might be slightlydifferent.
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 4 lines
Quoted 4 lines
2. no UC/WB/WC definition (CPU cachable) in
drm_xe_gem_mmap_offset,Quoted 9 lines
will it be handled by KMD automatically?
Quoted 3 lines
For example: if set 1-way, it means GPU could snoop CPU cache, we could use WB in mmap offset If it is
COHERENCY_NONE, we could only useUC, all these logic is handled by KMD automatically?
Yes, the mmap will use the same CPU side caching mode that you set at
gem_createwithsmem_caching(wb/wc/uc).It means it could not be changed after
gem_create. Sometimes, UMD doesn’t know Whether the data need to be accessed by CPU atgem_create. only application Knows the purpose of allocation & the surface usage. So, may need change UMD api to let applications set whether theQuoted 7 lines
Quoted 7 lines
resource is "accessible", or it will hurt the perf.
Right, the
smem_cachingis immutable.Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
3. about " For imported dma-buf (from a different device) the coherency mode is also implicit and must also be either 1WAY or
2WAY"
Quoted 19 lines
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
it means it must be 1way or 2way, and UMD need not to set it?
If you are going to
vm_bindit you need to supply thepat_indexand in this case it needs to be either 1way or 2way. The object comes from a different device so we might not have smem_caching/coh_mode like we do for native objects.Quoted 7 lines
How about the GPU cache setting of imported dma-buf?
Is there possible conflict between dma-buf producer and consumer? For example:
In producer side , one bo is set GPU L3 cache, also CPU cacheable, it should be 2Way In client side, we know the coherency should be 1-way or 2-way, if it suppose no GPU cache Used. It just set 1-way. It may cause some problems?Quoted 3 lines
If the producer and consumer are the same device, then I think the original gem_create.coh_mode is considered, since it's effectively still a native object. In which case the normal rules apply.
Quoted 17 lines
But the consumer does not know the
coh_mode, so , it is possible that it set will set Another value, then it will failed.
And because consumer does not know it is 1-way or 2-way, consumer could not Set the gpu cache correctly. Because if it is 1-way, cpu could not snoop gpu cacheQuoted 11 lines
If it comes from a different device then we don't know what gem_create.coh_mode is (or if that even exists) since we don't have the usual native object, but just some opaque dma-buf object. In this case it must be at least 1way or 2way.
From KMD pov we only care about userspace not being able to bypass
the clearing. There might still be other coherency issues though.Quoted 2 lines
Also , when a bo is created, UMD does not know whether it will be
exported.
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
It maybe be set as
COHERENCY_NONE, right?Yeah, you can use whatever you want. Just that your
smem_cachingwill need to be wc/uc if usingCOH_NONE. On the importer side, if it comes from the same device then it's just a normal native object and the normalrules apply.
Quoted 4 lines
Quoted 4 lines
Currently this would mean also using
pat_indexwithCOH_NONE. If it is imported from a different device then you need to use apat_indexwith 1way or 2way, since KMD doesn't really know how the object is mapped on the CPU or if the CPU caches are dirty (potentially withclearing).
Quoted 7 lines
Quoted 3 lines
Is this going to be problematic?
If imported surfaces is from same device, how consumer know it is original
coh_nonOr 1-way or 2-way?And it also means producer and consumer could have different
pat_indexforQuoted 9 lines
same bo.
Ok, if KMD instead allows 1way or 2way for dma-buf when it's external or from the same device? Would that help here? i.e you can always select 1way or 2way, even if it came from the same device and was
coh_none?Other option is you could call the
vm_bindioctl to check if thecoh_modeis supported by checking if it returns an error? But maybe that is too nasy.
Quoted 5 lines
Still a bit confuse, I create a bo with
coh_none, then export it to a PRIME fd. In consumer side, it does not know it iscoh_none, and will callvm_bindto setpat_index.You mean it will return a error when I set different coh value though
vm_bind? And trycoh_non, 1-way, 2-way, then get the correct one? TBH, it is ugly.
Yeah, it is nasty.
Quoted 6 lines
My concern is: if it is
coh_nonor 1-way, it means I could not use GPU cache if the bo also is CPU accessible. Because CPU could not snoop GPU cache.
So, GPU cache specified bypat_indexalso will be rejected?Could we add some query interface to query the CPU cache and coherency setting of a bo?
Say if the buffer comes from a different device + driver, what pat_index would you choose for that? There is no such thing as coh_mode/smem_caching for such an object. If you have to handle that case can't you just treat all imported dma-buf the same as that? The new proposal was to allow 1way or 2way for any dma-buf object, even if it comes from the same device and has different BO coh_mode. Do you need more than that? AFAICT the query would only really work for objects imported from the same device.
Quoted 1,008 lines
Quoted 1,008 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 10 lines
Quoted 10 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Wednesday, August 30, 2023 7:28 PM
To: Roper, Matthew D <[email protected]>
Cc: [email protected]; Mishra, PallaviQuoted 3 lines
Quoted 3 lines
Thomas Hellström <[email protected]>; Joonas Lahtinen <[email protected]>; De Marchi, Lucas <[email protected]>; Souza, Jose <[email protected]>;
Hazubski,
Quoted 2 lines
Quoted 2 lines
Filip <[email protected]>; Zhang, Carl
<[email protected]>; Yu,Effie
Quoted 3 lines
Quoted 3 lines
<[email protected]>
Subject: Re: [RFC 5/5] drm/xe/uapi: supportpat_indexselection withQuoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
vm_bindQuoted 6 lines
Quoted 6 lines · Matt Roper
Quoted 4 lines · Matthew Auld
Quoted 3 lines
Allow userspace to directly control the
pat_indexfor a given vm binding. This should allow directly controlling the coherency, caching and potentially other stuff in the future for theppGTT binding.
Quoted 37 lines
Quoted 37 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
The exact meaning behind the
pat_indexis very platform specific (see BSpec or PRMs) but effectively maps to some predefined memory attributes. From the KMD pov we only care about the coherency that is provided by thepat_index, which falls into eitherNONE, 1WAY or 2WAY.
Quoted 30 lines
Quoted 30 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The
vm_bindcoherency mode for the givenpat_indexneeds to matchthe
Quoted 27 lines
Quoted 27 lines
Quoted 10 lines
Quoted 2 lines
given
coh_modethat was set at object creation. For platforms that lackIs it actually important to match the coherency mode? I think one of the main goals was to know up front if userspace might be using a non-snooping PAT setting that would let it bypass the CPU cache (and potentially read old, stale data from a different process if the buffer's clear value is still sitting in cache and hasn't landed in memory yet).
Quoted 6 lines
If that's the only concern, then I think it should still be fine to map with a non-matching PAT as long as it's more coherent than the one specified at creation, right? E.g., if the buffer was created with 1-way coherency, it would be fine to map it with 2-way because userspace still can't use that to observe the previous contents of the buffer. Or
Yeah, I guess we could in theory do something that.
Quoted 7 lines
if the buffer was created with "non-coherent" then we've already done the necessary clflushing in kernel before handing to buffer over to userspace to ensure the clear value landed in memory, so any valid PAT index should be safe (from a security POV) after that, right? Any other problems that arise from mismatched coherency would just be contained to the app possibly shooting itself in the foot, which isn't really our concern.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
That is also my understanding, at least from the KMD security pov. If you allocate as wb then you must use at least 1way, since there is no flushing
for
Quoted 1 line
Quoted 1 line
clearing or swap-in. For uc/wc you could in theory use whatever you
want.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
the explicit coherency mode, we treat UC/WT/WC as NONE and
WB
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
as
2WAY.
Quoted 2 lines
Quoted 2 lines
For userptr mappings we lack a corresponding gem object, so the
expected
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
coherency mode is instead implicit and must fall into either 1WAY or 2WAY. Trying to use NONE will be rejected by the kernel. For imported dma-buf (from a different device) the coherency mode is also implicit and must also be either 1WAY or
2WAY.
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
As part of adding
pat_indexsupport withvm_bindwe also need stopusing
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
xe_cache_leveland instead use thepat_indexin various places. We still make use ofxe_cache_level, but only as a convenience for kernel internal objectsi (internally it maps to some reasonablepat_index). ForQuoted 12 lines
Quoted 3 lines
Quoted 3 lines
Maybe we should kill
xe_cache_levelcompletely and just assignxe_gt->pat_cached/xe_gt->pat_uncachedat init that can be used in appropriate places, similar to what we do with MOCS (gt-mocs.uc_index,
Quoted 8 lines
Quoted 1 line
gt->mocs.wb_index)?OK, seems reasonable to me.
Quoted 4 lines
Quoted 4 lines
now this is just a 1:1 conversion of the existing code, however for platforms like MTL+ we might need to give more control through
bo_createQuoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
or stop using WB on the CPU side if we need CPU access.
Bspec: 45101, 44235 #xe
Bspec: 70552, 71582, 59400 #xe2Signed-off-by Matthew Auld <[email protected]>Cc Pallavi Mishra <[email protected]>Cc Thomas Hellström <[email protected]>Cc Joonas Lahtinen <[email protected]>Cc Lucas De Marchi <[email protected]>Cc Matt Roper <[email protected]>Cc José Roberto de Souza <[email protected]>Cc Filip Hazubski <[email protected]>Cc Carl Zhang <[email protected]>Cc Effie Yu <[email protected]>---
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/i915/gt/intel_gtt.h | 2 +- drivers/gpu/drm/xe/tests/xe_migrate.c | 2 +- drivers/gpu/drm/xe/xe_ggtt.c | 7 ++- drivers/gpu/drm/xe/xe_ggtt_types.h | 2 +- drivers/gpu/drm/xe/xe_migrate.c | 14 ++--- drivers/gpu/drm/xe/xe_pt.c | 32 +++++------- drivers/gpu/drm/xe/xe_pt.h | 6 +-- drivers/gpu/drm/xe/xe_vm.c | 73+++++++++++++++++++++----
Quoted 6 lines
Quoted 6 lines
-
Quoted 5 lines
Quoted 5 lines
-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm_types.h | 13 +++-- include/uapi/drm/xe_drm.h | 41 ++++++++++++++- 10 files changed, 134 insertions(+), 58 deletions(-)Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Review
Quoted 1 line
(quoted diff) +0
-0
b/drivers/gpu/drm/i915/gt/intel_gtt.hQuoted 5 lines
drivers/gpu/drm/i915/gt/intel_gtt.h struct i915_address_space { · line 302
Quoted 5 lines
index
4d6296cdbcfd..bb4c182048c3100644drivers/gpu/drm/i915/gt/intel_gtt.h +0
-0@@ -302,7 +302,7 @@ struct i915_address_space { (*alloc_scratch_dma)(struct i915_address_space*vm,
Quoted 33 lines
Quoted 33 lines
int
Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Review
Quoted 9 lines
drivers/gpu/drm/i915/gt/intel_gtt.h +1
-1sz);u64 (*pte_encode)(dma_addr_t addr, -unsigned int pat_index, +u32 pat_index,u32 flags);/* Create a valid PTE */ #define PTE_READ_ONLY BIT(0)#define PTE_LM BIT(1)(quoted diff) +0
-0
b/drivers/gpu/drm/xe/tests/xe_migrate.cQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/tests/xe_migrate.c static void xe_migrate_sanity_test(struct
Quoted 4 lines
index
5c8d5e78d9bc..7a128fd20a29100644drivers/gpu/drm/xe/tests/xe_migrate.c +0
-0@@ -301,7 +301,7 @@ static void xe_migrate_sanity_test(struct
xe_migrateQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
*m, struct kunit *test)
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/tests/xe_migrate.c +0
-0/* First part of the test, are we updating our pagetable bo with a new
entry? */
Quoted 1 line
Quoted 1 line
xe_map_wr(xe, &bo->vmap, XE_PAGE_SIZE *(
NUM_KERNEL_PDE-Quoted 11 lines
Quoted 11 lines
1),
Quoted 9 lines
Quoted 9 lines
u64,
Quoted 3 lines
drivers/gpu/drm/xe/tests/xe_migrate.c
Quoted 3 lines
0xdeaddeadbeefbeef);drivers/gpu/drm/xe/tests/xe_migrate.c +1
-1- expected =xe_pte_encode (pt,0 ,XE_CACHE_WB ,0 ); + expected = xe_pte_encode(pt, 0, xe_pat_get_index(xe,
XE_CACHE_WB), 0);Quoted 4 lines
Review
Quoted 4 lines
if (m->q->vm->flags & XE_VM_FLAG_64K) expected |= XE_PTE_PS64; if (xe_bo_is_vram(pt))(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_ggtt.cQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c u64 xe_ggtt_pte_encode(struct xe_bo *bo,
Quoted 4 lines
index
209fa053d9fb..4134c26150a5100644drivers/gpu/drm/xe/xe_ggtt.c +0
-0@@ -41,7 +41,8 @@ u64 xe_ggtt_pte_encode(struct xe_bo *bo,u64
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
bo_offset)Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c u64 xe_ggtt_pte_encode(struct xe_bo *bo, · line 44
Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c +0
-1pte |=XE_GGTT_PTE_DM ;if ((ggtt->pat_encode).pte_encode) - pte = (ggtt->pat_encode).pte_encode(xe, pte,
XE_CACHE_WB_1_WAY);Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_ggtt.c u64 xe_ggtt_pte_encode(struct xe_bo *bo, · line 45
Quoted 2 lines
drivers/gpu/drm/xe/xe_ggtt.c +2
-0+ pte = (ggtt->pat_encode).pte_encode(xe, pte, +xe_pat_get_index(xe,
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
XE_CACHE_WB_1_WAY));Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c static void primelockdep(struct xe_ggtt
Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c +0
-0return pte; }@@ -102,10 +103,8 @@ static void primelockdep(struct xe_ggtt*ggtt)
Quoted 86 lines
Quoted 86 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c +0
-0} static u64 xelpg_ggtt_pte_encode_pat(struct xe_device *xe,u64
pte_pat,Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_ggtt.c
Quoted 1 line
drivers/gpu/drm/xe/xe_ggtt.c +0
-1-enum xe_cache_level cache)
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Review
Quoted 8 lines
drivers/gpu/drm/xe/xe_ggtt.c +1
-2+ u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELPG_GGTT_PTE_PAT_MASK );if (pat_index &BIT (0 ))(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_ggtt_types.hQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_ggtt_types.h struct xe_ggtt { · line 34
Quoted 8 lines
index
7e55fac1a8a9..0bc40cb072e3100644drivers/gpu/drm/xe/xe_ggtt_types.h +0
-1@@ -31,7 +31,7 @@ struct xe_ggtt {struct { u64 (*pte_encode)(struct xe_device *xe, u64 pte_pat, - enum xe_cache_levelcache);
Quoted 32 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Review
Quoted 5 lines
drivers/gpu/drm/xe/xe_ggtt_types.h +1
-0+ u32 pat_index); } pat_encode; };(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_migrate.cQuoted 16 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
index
a782ea282cb6..54585e98452a100644drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -24,6 +24,7 @@#include "xe_lrc.h" Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_migrate.c +1
-0#include "xe_map.h" #include "xe_mocs.h" +#include "xe_pat.h" #include "xe_pt.h" #include "xe_res_cursor.h" #include "xe_sched_job.h" @@ -162,6 +163,7 @@ static int xe_migrate_prepare_vm(struct
xe_tile*tile,
Quoted 2 lines
Quoted 2 lines
struct
xe_migrate*m,Quoted 1 line
Quoted 1 line
u32
num_entries=NUM_PT_SLOTS,num_level= vm-pt_root[id]-
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
level;
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct
Quoted 7 lines
u32 map_ofs, level, i; struct xe_bo *bo, *batch = tile->mem.kernel_bb_pool->bo;drivers/gpu/drm/xe/xe_migrate.c +1
-0+u32 pat_index =xe_pat_get_index (xe,XE_CACHE_WB );u64 entry;int ret;@@ -189,14 +191,14 @@ static int xe_migrate_prepare_vm(struct
xe_tileQuoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
*tile, struct
xe_migrate*m,Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 192
Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-1return ret; } - entry = xe_pde_encode(bo, bo->size - XE_PAGE_SIZE,
XE_CACHE_WB);Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 194
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ entry = xe_pde_encode(bo, bo->size - XE_PAGE_SIZE,
pat_index);Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 199
Quoted 7 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-1xe_pt_write (xe, &vm->pt_root[id]->bo->vmap,0 , entry); map_ofs = (num_entries - num_level) *XE_PAGE_SIZE ;/* Map the entire BO in our level 0 pt */ for (i =0 , level =0 ; i < num_entries; level++) { - entry = xe_pte_encode(bo, i * XE_PAGE_SIZE,
XE_CACHE_WB,Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
0);
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 201
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ entry = xe_pte_encode(bo, i * XE_PAGE_SIZE,
pat_index, 0);Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 202
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-0xe_map_wr(xe, &bo->vmap, map_ofs + level * 8, u64,entry);
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -214,7 +216,7 @@ static int xe_migrate_prepare_vm(struct
xe_tile*tile,
Quoted 3 lines
Quoted 3 lines
struct
xe_migrate*m,Quoted 2 lines
Quoted 2 lines
for (i = 0; i < batch->size; i += vm->flags & XE_VM_FLAG_64K ?Quoted 7 lines
Quoted 7 lines
XE_64K_PAGE_SIZE:Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 2 lines
XE_PAGE_SIZE) {drivers/gpu/drm/xe/xe_migrate.c +0
-1- entry = xe_pte_encode(batch, i,
XE_CACHE_WB, 0);Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ entry =xe_pte_encode (batch, i, pat_index,0 ); xe_map_wr(xe, &bo->vmap, map_ofs + level*
Quoted 5 lines
Quoted 5 lines
8, u64,
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
entry);drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -239,7 +241,7 @@ static int xe_migrate_prepare_vm(struct
xe_tile*tile,
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
struct
xe_migrate*m,Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 4 lines
flags =
XE_PDE_64K;entry = xe_pde_encode(bo,
map_ofs+ (level - 1) *drivers/gpu/drm/xe/xe_migrate.c +0
-1- XE_PAGE_SIZE,
XE_CACHE_WB);Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ XE_PAGE_SIZE, pat_index); xe_map_wr(xe, &bo->vmap, map_ofs +
XE_PAGE_SIZEQuoted 7 lines
Quoted 7 lines
* level,
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
u64,
Quoted 4 lines
Quoted 4 lines
entry | flags); }drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -247,7 +249,7 @@ static int xe_migrate_prepare_vm(struct
xe_tile*tile,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
struct
xe_migrate*m,Quoted 7 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 7 lines
/* Write PDE's that point to our BO. */ for (i = 0; i < num_entries - num_level; i++) { entry = xe_pde_encode(bo, i * XE_PAGE_SIZE,drivers/gpu/drm/xe/xe_migrate.c +1
-1- XE_CACHE_WB); + pat_index); xe_map_wr(xe, &bo->vmap, map_ofs +
XE_PAGE_SIZEQuoted 130 lines
Quoted 130 lines
+
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(i + 1) * 8, u64, entry); @@ -1256,7 +1258,7@@
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
xe_migrate_update_pgtables(struct
xe_migrateQuoted 5 lines
Quoted 5 lines
*m,
Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 4 lines
XE_WARN_ON(pt_bo->size != SZ_4K);drivers/gpu/drm/xe/xe_migrate.c +0
-1- addr = xe_pte_encode(pt_bo, 0,
XE_CACHE_WB, 0);Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ addr = xe_pte_encode(pt_bo, 0,xe_pat_get_index(xe,
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
XE_CACHE_WB), 0);Quoted 10 lines
Quoted 10 lines
bb->cs[bb->len++] = lower_32_bits(addr); bb->cs[bb->len++] = upper_32_bits(addr); }(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_pt.cindex64713f400d94..019af2920078100644drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -10,6 +10,7 @@#include "xe_gt.h" Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_pt.c +1
-0#include "xe_gt_tlb_invalidation.h" #include "xe_migrate.h" +#include "xe_pat.h" #include "xe_pt_types.h" #include "xe_pt_walk.h" #include "xe_res_cursor.h" @@ -57,24 +58,22 @@ static struct xe_pt *xe_pt_entry(struct
xe_pt_dir
*pt_dir, unsigned int index)Quoted 12 lines
Quoted 12 lines
* * Return: An encoded page directory entry. No errors. */drivers/gpu/drm/xe/xe_pt.c +2
-2-u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level cache) +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 +pat_index) {u64 pde;struct xe_vm *vm = bo->vm;struct xe_device *xe = vm->xe;Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c +1
-2- pde =xe_bo_addr (bo, bo_offset,XE_PAGE_SIZE ); pde |=XE_PAGE_PRESENT |XE_PAGE_RW ;if ((vm->pat_encode).pde_encode) - pde = (vm->pat_encode).pde_encode (xe, pde, cache); + pde = (vm->pat_encode).pde_encode(xe, pde,
pat_index);Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 9 lines
return pde; }drivers/gpu/drm/xe/xe_pt.c +1
-1-static u64 __pte_encode (u64 pte,enum xe_cache_level cache, +static u64 __pte_encode (u64 pte,u32 pat_index,struct xe_vma *vma,u32 pt_level) { struct xe_vm *vm = xe_vma_vm(vma); @@ -89,7 +88,7 @@static
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
u64 __pte_encode(u64 pte, enum
xe_cache_levelcache,Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 5 lines
pte |= XE_PTE_NULL; if ((vm->pat_encode).pte_encode)drivers/gpu/drm/xe/xe_pt.c +1
-1- pte = (vm->pat_encode).pte_encode (xe, pte, cache); + pte = (vm->pat_encode).pte_encode(xe, pte,
pat_index);Quoted 47 lines
Quoted 21 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c static u64 __pte_encode(u64 pte, enum
Quoted 4 lines
if (pt_level == 1) pte |= XE_PDE_PS_2M;drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -112,7 +111,7 @@ static u64 __pte_encode(u64 pte, enum
xe_cache_levelcache,Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c +0
-1* * Return: An encoded page-table entry. No errors. */ -u64 xe_pte_encode(struct xe_bo *bo, u64 offset, enum
xe_cache_levelQuoted 6 lines
Quoted 6 lines
cache,
Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c u64 xe_pte_encode(struct xe_bo *bo, u64
Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c +1
-0+u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 pat_index,u32 pt_level) {u64 pte;@@ -121,7 +120,7 @@ u64 xe_pte_encode(struct xe_bo *bo, u64offset,
Quoted 4 lines
Quoted 4 lines
enum
xe_cache_levelcache,Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c +0
-0if (xe_bo_is_vram (bo) ||xe_bo_is_stolen_devmem (bo)) pte |=XE_PPGTT_PTE_DM ;Quoted 19 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c +1
-1-return __pte_encode (pte, cache,NULL , pt_level); +return __pte_encode (pte, pat_index,NULL , pt_level); } static u64 __xe_pt_empty_pte(struct xe_tile *tile, struct
xe_vm*vm, @@ -134,12 +133,12 @@ static u64
__xe_pt_empty_pte(structxe_tile*tile,
Quoted 11 lines
Quoted 11 lines
struct
xe_vm*vm,Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 5 lines
if (level == 0) { u64 empty = xe_pte_encode(vm->scratch_bo[id], 0,drivers/gpu/drm/xe/xe_pt.c +1
-1- XE_CACHE_WB, 0); + xe_pat_get_index(vm->xe,
XE_CACHE_WB), 0);Quoted 4 lines
Quoted 4 lines
return empty; } else { return xe_pde_encode(vm->scratch_pt[id][level - 1]-bo, 0,
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 2 lines
drivers/gpu/drm/xe/xe_pt.c +1
-1- XE_CACHE_WB); + xe_pat_get_index(vm->xe,
XE_CACHE_WB));Quoted 3 lines
Quoted 3 lines
} }Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c struct xe_pt_stage_bind_walk { · line 370
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c +0
-2@@ -368,8 +367,6 @@ struct xe_pt_stage_bind_walk {struct xe_vm *vm;/** @tile: The tile we're building for. */ struct xe_tile *tile; -/** @cache: Desired cache level for the ptes */ -enum xe_cache_level cache;/** @default_pte: PTE flag only template. No address is associated */
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +1
-1u64 default_pte; /** @dma_offset: DMA offset to add to the PTE. */ @@ -604 ,7 +601,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent,Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
pgoff_toffset,Quoted 3 lines
Quoted 3 lines
pte = __pte_encode(is_null ? 0 : xe_res_dma(curs) + xe_walk-
dma_offset,Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +0
-1- xe_walk->cache, xe_walk->vma,level);
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +1
-0+ xe_walk->vma->pat_index,xe_walk-
Quoted 64 lines
Quoted 4 lines
Quoted 4 lines
vma,
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
level);
Quoted 2 lines
Quoted 2 lines
pte |= xe_walk->default_pte;Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind_entry(struct xe_ptw · line 666
Quoted 3 lines
/*drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -669,7 +666,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent,
pgoff_toffset,Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind_entry(struct xe_ptw · line 673
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +0
-1xe_child->is_compact = true; } - pte = xe_pde_encode(xe_child->bo, 0, xe_walk-cache)
Quoted 51 lines
|
Quoted 4 lines
Quoted 4 lines
flags;
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind_entry(struct xe_ptw · line 670
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +1
-0+ pte = xe_pde_encode(xe_child->bo, 0, xe_walk->vma-
pat_index) | flags;Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +0
-0ret = xe_pt_insert_entry(xe_walk, xe_parent, offset,
xe_child,Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +0
-0pte);}drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -730,13 +727,8 @@ xe_pt_stage_bind(struct xe_tile *tile,struct
xe_vmaQuoted 2 lines
Quoted 2 lines
*vma,
Quoted 1 line
Quoted 1 line
if (vma && vma->gpuva.flags &
XE_VMA_ATOMIC_PTE_BIT)Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
xe_walk.default_pte |=
XE_USM_PPGTT_PTE_AE;Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 5 lines
Quoted 1 line
xe_walk.dma_offset = vram_region_gpu_offset(bo-
ttm.resource);
drivers/gpu/drm/xe/xe_pt.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c +0
-3- xe_walk.cache =XE_CACHE_WB ; - } else { - if (!xe_vma_has_no_bo(vma) && bo->flags &
XE_BO_SCANOUT_BIT)Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c +1
-3- xe_walk.cache =XE_CACHE_WT ; - else - xe_walk.cache = XE_CACHE_WB; } +if (!xe_vma_has_no_bo (vma) &&xe_bo_is_stolen (bo)) xe_walk.dma_offset =xe_ttm_stolen_gpu_offset(xe_bo_device(bo));
Quoted 8 lines
Quoted 8 lines
(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_pt.hindex01be7ab08f87..1d433a5a96b4100644drivers/gpu/drm/xe/xe_pt.h +0
-0@@ -45,10 +45,8 @@ __xe_pt_unbind_vma(struct xe_tile *tile,struct
xe_vma*vma, structxe_exec_queuQuoted 11 lines
Quoted 11 lines
bool xe_pt_zap_ptes(struct
xe_tile*tile, structxe_vma
*vma);drivers/gpu/drm/xe/xe_pt.h +2
-3-u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level level); +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 +pat_index); -u64 xe_pte_encode(struct xe_bo *bo, u64 offset, enum
xe_cache_levelcache,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.h
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.h +2
-1- u32 pt_level); +u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 pat_index, +u32
pt_level);Quoted 4 lines
Quoted 4 lines
#endif(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_vm.cindex
7eeeed0411f3..34603a7e84b0Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
100644
drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -6,6 +6,7 @@#include "xe_vm.h" Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +1
-0#include <linux/dma-fence-array.h> +#include <linux/nospec.h> #include <drm/drm_print.h> #include <drm/ttm/ttm_execbuf_util.h> @@ -874,7 +875,8 @@static struct
xe_vma*xe_vma_create(struct
xe_vmQuoted 1 line
Quoted 1 line
*vm,
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c static struct xe_vma
Quoted 10 lines
u64 start, u64 end, bool read_only, bool is_null,drivers/gpu/drm/xe/xe_vm.c +2
-1- u8 tile_mask) + u8 tile_mask, + u32 pat_index) {struct xe_vma *vma;struct xe_tile *tile;@@ -913,6 +915,8 @@ static struct xe_vma*xe_vma_create(struct
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
xe_vmQuoted 10 lines
Quoted 10 lines
*vm,
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c static void
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c +2
-0vma->tile_mask |=0x1 << id; } + vma->pat_index = pat_index; +if (vm->xe->info.platform ==XE_PVC ) vma->gpuva.flags |=XE_VMA_ATOMIC_PTE_BIT ;@@ -1194,10 +1198,8 @@ static voidxe_vma_op_work_func(struct
Quoted 25 lines
Quoted 25 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
work_struct*w);Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0static void vm_destroy_work_func (struct work_struct *w); static u64 xelp_ppgtt_pde_encode_pat(struct xe_device *xe,u64
pde_pat,Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-1-enum xe_cache_level cache)
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c static u64
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c +1
-2+ u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pde_pat &= ~(XELP_PDE_PAT_MASK );if (pat_index &BIT (0 ))@@ -1213,10 +1215,8 @@ static u64xelp_ppgtt_pde_encode_pat(struct
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
xe_device*xe, u64pde_pat,Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0} static u64 xelp_ppgtt_pte_encode_pat(struct xe_device *xe,u64
pte_pat,Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-1-enum xe_cache_level cache)
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c struct xe_vm *xe_vm_lookup(struct
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c +1
-2+ u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELP_PTE_PAT_MASK );if (pat_index &BIT (0 ))@@ -1622,7 +1622,7 @@ struct xe_vm *xe_vm_lookup(struct
xe_fileQuoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
*xef,
Quoted 2 lines
Quoted 2 lines
u32 id)
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0u64 xe_vm_pdp4_descriptor(struct xe_vm *vm, struct xe_tile*tile)
Quoted 71 lines
Quoted 71 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c struct xe_vm *xe_vm_lookup(struct · line 1625
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1{ return xe_pde_encode(vm->pt_root[tile->id]->bo, 0, -XE_CACHE_WB ); + xe_pat_get_index(vm->xe,
XE_CACHE_WB));Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c static void print_op(struct xe_device · line 2311
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0} static struct dma_fence *@@ -2311,7 +2311,7 @@ static void print_op(struct xe_device *xe,struct
Quoted 4 lines
Quoted 4 lines
drm_gpuva_op*op)Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0static struct drm_gpuva_ops *vm_bind_ioctl_ops_create (struct xe_vm *vm,struct xe_bo *bo, u64 bo_offset_or_userptr, u64 addr, u64range,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c static void print_op(struct xe_device · line 2315
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- u32 operation, u8 tile_mask, u32 region) + u32 operation, u8 tile_mask, u32 region, u32
pat_index)Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0{struct drm_gem_object *obj = bo ? &bo->ttm.base :NULL ;struct ww_acquire_ctx ww; @@ -2339,6 +2339,7 @@vm_bind_ioctl_ops_create(struct
xe_vm*vm,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
struct
xe_bo*bo,Quoted 1 line
Quoted 1 line
struct
xe_vma_op*op =
gpuva_op_to_vma_op(__op);Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 5 lines
op->tile_mask = tile_mask;drivers/gpu/drm/xe/xe_vm.c +1
-0+ op->pat_index = pat_index; op->map.immediate = operation &
XE_VM_BIND_FLAG_IMMEDIATE;Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c vm_bind_ioctl_ops_create(struct xe_vm
Quoted 2 lines
op->map.read_only =drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2366,6 +2367,7 @@ vm_bind_ioctl_ops_create(struct xe_vm*vm,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
struct
xe_bo*bo,Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0struct xe_vma_op *op =
gpuva_op_to_vma_op(__op);Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c vm_bind_ioctl_ops_create(struct xe_vm
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c +1
-0op->tile_mask = tile_mask; + op->pat_index = pat_index; op->prefetch.region = region; }break ;@@ -2408,7 +2410,8 @@ vm_bind_ioctl_ops_create(struct xe_vm*vm,
Quoted 24 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
struct
xe_bo*bo,Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c vm_bind_ioctl_ops_create(struct xe_vm · line 2412
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0} static struct xe_vma *new_vma(struct xe_vm *vm, struct
drm_gpuva_op_map*op,Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c vm_bind_ioctl_ops_create(struct xe_vm · line 2416
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c +2
-1- u8 tile_mask, bool read_only, bool is_null) + u8 tile_mask, bool read_only, bool is_null, + u32 pat_index) { struct xe_bo *bo = op->gem.obj ? gem_to_xe_bo(op-gem.obj) :
Quoted 13 lines
NULL;
Quoted 12 lines
Quoted 12 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0struct xe_vma *vma;@@ -2425,7 +2428,7 @@ static struct xe_vma *new_vma(struct
xe_vm*vm, struct
drm_gpuva_op_map*op,Quoted 8 lines
Quoted 8 lines
vma = xe_vma_create(vm, bo, op->gem.offset, op->va.addr, op->va.addr + op->va.range - 1, read_only, is_null,drivers/gpu/drm/xe/xe_vm.c +1
-1- tile_mask); + tile_mask, pat_index);if (bo)xe_bo_unlock (bo, &ww);Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c static int
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2539,7 +2542,7 @@ static intvm_bind_ioctl_ops_parse(struct
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
xe_vmQuoted 3 lines
Quoted 3 lines
*vm, struct
xe_exec_queue*q,Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0vma = new_vma(vm, &op-base.map,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0op->tile_mask, op-Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
map.read_only,
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2544
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- op->map.is_null); + op->map.is_null, op-
pat_index);Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c static int
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2567,7 +2570,7 @@ static intvm_bind_ioctl_ops_parse(struct
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
xe_vmQuoted 5 lines
Quoted 5 lines
*vm, struct
xe_exec_queue*q,Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2570
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0vma = new_vma(vm, op-base.remap.prev,
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0op->tile_mask,Quoted 8 lines
Quoted 8 lines
read_only,Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2572
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- is_null); + is_null, op-
pat_index);Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c static int
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2603,7 +2606,7 @@ static intvm_bind_ioctl_ops_parse(struct
Quoted 22 lines
Quoted 22 lines
Quoted 6 lines
Quoted 6 lines
xe_vmQuoted 5 lines
Quoted 5 lines
*vm, struct
xe_exec_queue*q,Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2606
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0vma = new_vma(vm, op-base.remap.next,
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0op->tile_mask,
read_only,Quoted 15 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2608
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- is_null); + is_null, op-
pat_index);Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -3158,8 +3161,14 @@ static intvm_bind_ioctl_check_args(struct
xe_device*xe,Quoted 6 lines
Quoted 6 lines
u32 obj = (*bind_ops)[i].obj; u64 obj_offset = (*bind_ops)[i].obj_offset; u32 region = (*bind_ops)[i].region;drivers/gpu/drm/xe/xe_vm.c +1
-0+u32 pat_index = (*bind_ops)[i].pat_index;bool is_null = op &XE_VM_BIND_FLAG_NULL ;Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+ if (XE_IOCTL_DBG(xe, pat_index >= xe-info.pat_table_n_entries)) {
drivers/gpu/drm/xe/xe_vm.c
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +4
-0+ err = -EINVAL ; + goto free_bind_ops; + } +if (i ==0 ) { *async = !!(op &
XE_VM_BIND_FLAG_ASYNC);Quoted 40 lines
Quoted 40 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
} else if (XE_IOCTL_DBG(xe, !*async) || @@ -3346,8+3355,25
Quoted 15 lines
Quoted 14 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
@@ int xe_vm_bind_ioctl(struct
drm_device*dev,
Quoted 12 lines
Quoted 12 lines
void *data, struct
drm_file*file)Quoted 11 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 11 lines
struct drm_gem_object *gem_obj; u64 range = bind_ops[i].range; u64 addr = bind_ops[i].addr;drivers/gpu/drm/xe/xe_vm.c +6
-0+u32 op = bind_ops[i].op;u32 obj = bind_ops[i].obj;u64 obj_offset = bind_ops[i].obj_offset; +u32 pat_index = bind_ops[i].pat_index; +u16 coh_mode; + + pat_index = array_index_nospec(pat_index, + xe-info.pat_table_n_entries);
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+ coh_mode = xe_pat_index_get_coh_mode(xe,
pat_index);Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+if (XE_IOCTL_DBG (xe, !coh_mode)) {Assuming we drop the unusable entries from the TGL table, this should be impossible, right? Any index that makes it past the
n_entriescheck at the top of the function should have a valid, non-zero
coh_modevalue.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
So this should probably be an assertion (to highlight a KMD bug) rather than just a silent uapi failure return.
Makes sense.
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c +5
-0+ err = -EINVAL ; + goto put_obj; + } + + if (XE_IOCTL_DBG(xe, VM_BIND_OP(op) ==
XE_VM_BIND_OP_MAP_USERPTR&&Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+ coh_mode ==
XE_GEM_COHERENCY_NONE))Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
{
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c +3
-0+ err = -EINVAL ; + goto put_obj; + }if (!obj)continue ;@@ -3375,6 +3401,22 @@ int xe_vm_bind_ioctl(struct
drm_deviceQuoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
*dev,
Quoted 13 lines
Quoted 13 lines
void *data, struct
drm_file*file)Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3406
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +3
-0goto put_obj; } } + +if (bos[i]->coh_mode) { + if (XE_IOCTL_DBG(xe, bos[i]->coh_mode !=
coh_mode)) {Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3410
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +4
-0+ err = -EINVAL ; + goto put_obj; + } + } else if (XE_IOCTL_DBG(xe, coh_mode ==
XE_GEM_COHERENCY_NONE)) {Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3412
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +2
-0+/* +* Imported dma-buf from a different device should
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3413
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+* require 1way or 2way coherency since we don't
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
know
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3414
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+* how it was mapped on CPU. Just assume is it
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c +5
-0+* potentially cached on CPU side. +*/ + err = -EINVAL ; + goto put_obj; + } }if (args->num_syncs) {@@ -3412,10 +3454,11 @@ int xe_vm_bind_ioctl(struct
drm_deviceQuoted 39 lines
Quoted 39 lines
Quoted 8 lines
Quoted 8 lines
*dev,
Quoted 7 lines
Quoted 7 lines
void *data, struct
drm_file*file)Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3459
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +1
-0u64 obj_offset = bind_ops[i].obj_offset;u8 tile_mask = bind_ops[i].tile_mask;u32 region = bind_ops[i].region; +u32 pat_index = bind_ops[i].pat_index; ops[i] = vm_bind_ioctl_ops_create(vm, bos[i],
obj_offset,Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3460
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0addr, range, op,
tile_mask,Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
Review
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- region); + region, pat_index);if (IS_ERR (ops[i])) { err =PTR_ERR (ops[i]); ops[i] =NULL ;(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_vm_types.hQuoted 7 lines
Quoted 7 lines
index
83a1f87b6537..508679ed3c74100644drivers/gpu/drm/xe/xe_vm_types.h +0
-0@@ -111,6 +111,11 @@ struct xe_vma {*/ u8 tile_present;Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vma { · line 115
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm_types.h +2
-0+/** +* @pat_index: The pat index to use when encoding the PTEs for
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vma { · line 116
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +1
-0+thisvma.
Quoted 9 lines
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm_types.h +3
-0+*/ +u32 pat_index; +struct {struct list_head rebind_link; } notifier;@@ -338,10 +343,8 @@ struct xe_vm {bool batch_invalidate_tlb;Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 342
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm_types.h +0
-2struct { -u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat, - enum xe_cache_levelcache);
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 343
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +0
-1- u64 (*pde_encode)(struct xe_device *xe, u64
pde_pat,Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 344
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +0
-1-enum xe_cache_level cache);
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 346
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +1
-0+ u64 (*pte_encode)(struct xe_device *xe, u64 pte_pat,u32
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
pat_index);Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 347
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +1
-0+ u64 (*pde_encode)(struct xe_device *xe, u64
pde_pat,Quoted 40 lines
Quoted 40 lines
u32
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
pat_index);Quoted 21 lines
Quoted 21 lines
drivers/gpu/drm/xe/xe_vm_types.h +2
-0} pat_encode; };@@ -417,6 +420,8 @@ struct xe_vma_op {struct async_op_fence *fence;/** @tile_mask: gt mask for this operation */ u8 tile_mask; +/** @pat_index: The pat index to use for this operation. */ +u32 pat_index;/** @flags: operation flags */ enum xe_vma_op_flags flags;(quoted diff) +0
-0
b/include/uapi/drm/xe_drm.hindex64bc66d4b550..0c15b6f32447100644include/uapi/drm/xe_drm.h +0
-0@@ -600,8 +600,45 @@ struct drm_xe_vm_bind_op {*/ __u32 obj;Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 604
Quoted 4 lines
include/uapi/drm/xe_drm.h +2
-2-/** @pad: MBZ */ -__u32 pad; +/** +* @pat_index: The platform defined @pat_index to use for this
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
mapping.
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 605
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* The index basically maps to some predefined memory attributes,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 607
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * including things like caching, coherency and likely other +stuffin
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 609
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * the future. The exact meaning of the pat_index is +platform specificBTW, "other stuff in the future" already includes compression on Xe2, we just haven't landed the patches for the Xe2 table yet.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Ok, good to know.
Quoted 2 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 610
Quoted 2 lines
include/uapi/drm/xe_drm.h +1
-0+* and defined in the Bspec and PRMs. When the KMD sets up
Quoted 85 lines
Quoted 85 lines
the
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
binding
Quoted 4 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 614
Quoted 4 lines
include/uapi/drm/xe_drm.h +4
-0+* the index here is encoded into the ppGTT PTE. +* +* For coherency the @pat_index needs to match the +* drm_xe_gem_create.coh_mode, so either
XE_GEM_COHERENCY_NONE,Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 615
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* XE_GEM_COHERENCY_1WAY or
XE_GEM_COHERENCY_2WAY. TheQuoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
KMD will extract
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 616
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* the coherency mode from the @pat_index and reject if there is a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 619
Quoted 3 lines
include/uapi/drm/xe_drm.h +3
-0+* mismatch (see note below for pre-MTL platforms). +* +* Note: On pre-MTL platforms there is only a caching mode and no
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 620
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* explicit coherency mode, but on such hardware there is always a
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 622
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * shared-LLC (or is dgpu) so all GT memory accesses are +coherentwith
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 623
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* CPU caches even with the caching mode set as uncached. It's
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 624
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+onlythe
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 625
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* display engine that is incoherent (on dgpu it must be in VRAM which
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 626
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* is always mapped as WC on the CPU). However to keep the uapi
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
somewhat
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 627
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* consistent with newer platforms the KMD groups the different
Quoted 13 lines
Quoted 13 lines
cache
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 628
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* levels into the following coherency buckets on all pre-MTL platforms:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 633
Quoted 5 lines
include/uapi/drm/xe_drm.h +5
-0+* +* ppGTT UC -> XE_GEM_COHERENCY_NONE +* ppGTT WC -> XE_GEM_COHERENCY_NONE +* ppGTT WT -> XE_GEM_COHERENCY_NONE +* ppGTT WB -> XE_GEM_COHERENCY_2WAY As noted on the previous patch, it seems like 2-way is appropriate for LLC platforms, but 1-way might be a more accurate description of dGPU behavior.
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 636
Quoted 3 lines
include/uapi/drm/xe_drm.h +3
-0+* +* In practice UC/WC/WT should only ever used for scanout +surfaces on
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 637
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* such platforms since it is only the display engine that is actually
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 639
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * incoherent. Everything else should typically use WB given +that weWhat if we're sharing our buffers with some other (non-GPU) device? Are there cases where that other device wouldn't be coherent with the LLC, so we'd want to use one of these?
Yeah, I guess there might be cases like that. I'll reword.
Quoted 2 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Matt
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 640
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* have a shared-LLC. On MTL+ this completely changes (also potentially
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 641
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* no shared-LLC) and the HW defines the coherency mode as part
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 642
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+ofthe
Quoted 3 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op {
Quoted 3 lines
include/uapi/drm/xe_drm.h +3
-0+* @pat_index. +* +* Note: For userptr and externally imported dma-buf the kernel
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
expects
Quoted 10 lines
Quoted 9 lines
include/uapi/drm/xe_drm.h +3
-1+* either 1WAY or 2WAY for the @pat_index. +*/ +__u32 pat_index;union {/** -- 2.41.0
Re: [Intel-xe] [RFC 5/5] drm/xe/uapi: support pat_index selection with vm_bind
Quoted 216 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Tuesday, September 5, 2023 5:47 PMQuoted 127 lines · Carl
Quoted 127 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Monday, September 4, 2023 5:24 PMQuoted 9 lines · Carl
Quoted 9 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Thursday, August 31, 2023 6:44 PMQuoted 2 lines · Carl
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Thursday, August 31, 2023 12:02 AMQuoted 2 lines · Carl
Several questions:
1. thepat_indexfromvm_bindwill override the setting from
bo_create?Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
How to keep the value from
bo_createunchanged?You only set the
pat_indexduringvm_bind. Atgem_createyou just need tell the kernel what CPU side caching mode to use for system memory (wb/wc/uc), plus the expected GPU coherency mode of thepat_index. The current expectation is that the coherency mode of thepat_indexshould match the one atgem_create.Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
It means that we set the incomplete
pat_indexatgem_create(coherency
only) And set another part atvm_bind(gpu cache) Why we separate them? Why not set a full value atvm_bind?You always give the full
pat_indexatvm_bind. It's platform specific but the BSpec for the most part tells you what the coherency mode of the
pat_indexis.Quoted 4 lines
Quoted 4 lines
For example if you look at the MTL
pat_indextable it tells you thecoh_mode:Quoted 8 lines
Quoted 8 lines
pat_index0-2 =coh_mode"No snoop" =COH_NONEpat_index 3 = coh_mode "1way" = COH_1WAY pat_index 4 = coh_mode "2way" = COH_2WAYOn future platforms the table is larger and also encodes stuff like compression, but KMD really only cares about the
coh_mode, since that will restrict the allowed CPUsmem_cachingvalues that userspace canselect.
Quoted 4 lines
Quoted 4 lines
From KMD pov this matters since we always give userspace zeroed
memory, which is useful for normal applications but is also needed forsecurity reasons.
Quoted 11 lines
Quoted 11 lines
However the actual clearing might result in the writes only being visible in the CPU cache (not flushed to main memory), and if the GPU doesn't snoop the CPU cache it can read directly from main memory, which effectively bypasses the clearing. To prevent that the KMD needs to know how userspace will map the memory via the GPU i.e what is the
coh_modefor thepat_index. The gist is that if you selectCOH_NONEthen it needs to clflush the pages before giving them to userspace, which is done by forcing you to use uc/wc for thesmem_caching(the kernel will internally issue cflush on x86 when marking the pages as uc/wc). And if you select wb there is no flushing, buthere you need to use 1way or 2way.
Quoted 5 lines
Quoted 1 line
It sounds like a perf optimization, if no considering the perf,
you could always call Clflush to flush the cpu cache to main memory before giving it to user spaceI think the other thing is that the KMD is no longer doing any manual flushing, but instead the core kernel will now do that for us when marking the pages as wc/uc on x86.
Quoted 6 lines
Quoted 6 lines
AFAIK separating them is just to prevent userspace from mixing different coherency modes for the same object by having it immutable (you can still use different
pat_indexbut the coherency must becompatible).
Quoted 6 lines
Quoted 6 lines
Also at creation time we can determine if the
smem_cachingis compatible with thecoh_mode.Note that it might be that we ditch the
COH_2WAYand just haveCOH_NONEandCOH_AT_LEAST_1WAY, in which case this might be slightlydifferent.
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 4 lines
Quoted 4 lines
2. no UC/WB/WC definition (CPU cachable) in
drm_xe_gem_mmap_offset,Quoted 9 lines
will it be handled by KMD automatically?
Quoted 3 lines
For example: if set 1-way, it means GPU could snoop CPU cache, we could use WB in mmap offset If it is
COHERENCY_NONE, we could only useUC, all these logic is handled by KMD automatically?
Yes, the mmap will use the same CPU side caching mode that you set at
gem_createwithsmem_caching(wb/wc/uc).It means it could not be changed after
gem_create. Sometimes, UMD doesn’t know Whether the data need to be accessed by CPU atgem_create. only application Knows the purpose of allocation & the surface usage. So, may need change UMD api to let applications set whether theQuoted 7 lines
Quoted 7 lines
resource is "accessible", or it will hurt the perf.
Right, the
smem_cachingis immutable.Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
3. about " For imported dma-buf (from a different device) the coherency mode is also implicit and must also be either 1WAY or
2WAY"
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
it means it must be 1way or 2way, and UMD need not to set it?
If you are going to
vm_bindit you need to supply thepat_indexand in this case it needs to be either 1way or 2way. The object comes from a different device so we might not have smem_caching/coh_mode like we do for native objects.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
How about the GPU cache setting of imported dma-buf?
Is there possible conflict between dma-buf producer and consumer? For example:
In producer side , one bo is set GPU L3 cache, also CPU cacheable, it should be 2Way In client side, we know the coherency should be 1-way or 2-way, if it suppose no GPU cache Used. It just set 1-way.
Quoted 43 lines
Quoted 43 lines
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
It may cause some problems?
If the producer and consumer are the same device, then I think the original gem_create.coh_mode is considered, since it's effectively still a native object. In which case the normal rules apply.
Quoted 17 lines
But the consumer does not know the
coh_mode, so , it is possible that it set will set Another value, then it will failed. And because consumer does not know it is 1-way or 2-way, consumer could not Set the gpu cache correctly. Because if it is 1-way, cpu could not snoop gpu cacheQuoted 11 lines
If it comes from a different device then we don't know what gem_create.coh_mode is (or if that even exists) since we don't have the usual native object, but just some opaque dma-buf object. In this case it must be at least 1way or 2way.
From KMD pov we only care about userspace not being able to
bypass the clearing. There might still be other coherency issues though.Quoted 2 lines
Also , when a bo is created, UMD does not know whether it will be
exported.
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
It maybe be set as
COHERENCY_NONE, right?Yeah, you can use whatever you want. Just that your
smem_cachingwill need to be wc/uc if usingCOH_NONE. On the importer side, if it comes from the same device then it's just a normal native object and the normalrules apply.
Quoted 5 lines
Quoted 5 lines
Currently this would mean also using
pat_indexwithCOH_NONE. If it is imported from a different device then you need to use apat_indexwith 1way or 2way, since KMD doesn't really know how the object is mapped on the CPU or if the CPU caches are dirty (potentially withclearing).
Quoted 6 lines
Quoted 3 lines
Is this going to be problematic?
If imported surfaces is from same device, how consumer know it is original
coh_nonOr 1-way or 2-way?Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
And it also means producer and consumer could have different
pat_indexforsame bo.
Ok, if KMD instead allows 1way or 2way for dma-buf when it's external or from the same device? Would that help here? i.e you can always select 1way or 2way, even if it came from the same device and was
coh_none?Quoted 3 lines
Quoted 3 lines
Other option is you could call the
vm_bindioctl to check if thecoh_modeis supported by checking if it returns an error? But maybe that istoo nasy.
Quoted 4 lines
Quoted 1 line
Still a bit confuse, I create a bo with
coh_none, then export it to a PRIME fd. In consumer side, it does not know it iscoh_none, and will callvm_bindtoset
pat_index.Quoted 2 lines
You mean it will return a error when I set different coh value though
vm_bind?Quoted 1 line
And try
coh_non, 1-way, 2-way, then get the correct one? TBH, it is ugly.Yeah, it is nasty.
Quoted 2 lines
My concern is: if it is
coh_nonor 1-way, it means I could not use GPU cache ifthe bo also is CPU accessible.
Quoted 4 lines
Because CPU could not snoop GPU cache.
So, GPU cache specified bypat_indexalso will be rejected?Could we add some query interface to query the CPU cache and coherency
setting of a bo?
Quoted 8 lines
Say if the buffer comes from a different device + driver, what
pat_indexwould you choose for that? There is no such thing as coh_mode/smem_caching for such an object. If you have to handle that case can't you just treat all imported dma-buf the same as that? The new proposal was to allow 1way or 2way for any dma-buf object, even if it comes from the same device and has different BOcoh_mode. Do you need more than that? AFAICT the query would only really work for objects imported from the same device.
Looks I misunderstood the proposal, so, you mean if it shared in one device, Consumer could choose 1-way ,2-way , any one should work , and no error returned, no rejection from vm_bind, right?
Quoted 1,028 lines
Quoted 59 lines
Quoted 59 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 10 lines
Quoted 10 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Wednesday, August 30, 2023 7:28 PM
To: Roper, Matthew D <[email protected]>
Cc: [email protected]; Mishra, PallaviQuoted 3 lines
Quoted 3 lines
Thomas Hellström <[email protected]>; Joonas Lahtinen <[email protected]>; De Marchi, Lucas <[email protected]>; Souza, Jose <[email protected]>;
Hazubski,
Quoted 2 lines
Quoted 2 lines
Filip <[email protected]>; Zhang, Carl
<[email protected]>; Yu,Effie
Quoted 3 lines
Quoted 3 lines
<[email protected]>
Subject: Re: [RFC 5/5] drm/xe/uapi: supportpat_indexselection withQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
vm_bindQuoted 7 lines
Quoted 7 lines · Matt Roper
Quoted 5 lines · Matthew Auld
Quoted 4 lines
Allow userspace to directly control the
pat_indexfor a given vm binding. This should allow directly controlling the coherency, caching and potentially other stuff in the future for theppGTT binding.
Quoted 29 lines
Quoted 29 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
The exact meaning behind the
pat_indexis very platform specific (see BSpec or PRMs) but effectively maps to some predefined memory attributes. From the KMD pov we only care about the coherency that is provided by thepat_index, which falls into eitherNONE, 1WAY or 2WAY.
Quoted 22 lines
Quoted 22 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The
vm_bindcoherency mode for the givenpat_indexneeds to matchthe
Quoted 19 lines
Quoted 19 lines
Quoted 10 lines
Quoted 2 lines
given
coh_modethat was set at object creation. For platforms that lackIs it actually important to match the coherency mode? I think one of the main goals was to know up front if userspace might be using a non-snooping PAT setting that would let it bypass the CPU cache (and potentially read old, stale data from a different process if the buffer's clear value is still sitting in cache and hasn't landed in memory yet).
Quoted 6 lines
If that's the only concern, then I think it should still be fine to map with a non-matching PAT as long as it's more coherent than the one specified at creation, right? E.g., if the buffer was created with 1-way coherency, it would be fine to map it with 2-way because userspace still can't use that to observe the previous contents of the buffer. Or
Yeah, I guess we could in theory do something that.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
if the buffer was created with "non-coherent" then we've already done the necessary clflushing in kernel before handing to buffer over to userspace to ensure the clear value landed in memory, so any valid PAT index should be safe (from a security POV) after that, right? Any other problems that arise from mismatched coherency would just be contained to the app possibly shooting itself in the foot, which isn't really our
concern.
Quoted 450 lines
Quoted 450 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
That is also my understanding, at least from the KMD security pov. If you allocate as wb then you must use at least 1way, since there is no flushing
for
Quoted 2 lines
Quoted 2 lines
clearing or swap-in. For uc/wc you could in theory use whatever you
want.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
the explicit coherency mode, we treat UC/WT/WC as NONE and
WB
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
as
2WAY.
Quoted 3 lines
Quoted 3 lines
For userptr mappings we lack a corresponding gem object, so the
expected
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
coherency mode is instead implicit and must fall into either 1WAY or 2WAY. Trying to use NONE will be rejected by the kernel. For imported dma-buf (from a different device) the coherency mode is also implicit and must also be either 1WAY or
2WAY.
Quoted 38 lines
Quoted 38 lines
Quoted 38 lines
Quoted 38 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
As part of adding
pat_indexsupport withvm_bindwe also need stopusing
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
xe_cache_leveland instead use thepat_indexin various places. We still make use ofxe_cache_level, but only as a convenience for kernel internal objectsi (internally it maps to some reasonablepat_index). ForQuoted 13 lines
Quoted 4 lines
Quoted 4 lines
Maybe we should kill
xe_cache_levelcompletely and just assignxe_gt->pat_cached/xe_gt->pat_uncachedat init that can be used in appropriate places, similar to what we do with MOCS (gt-mocs.uc_index,
Quoted 8 lines
Quoted 1 line
gt->mocs.wb_index)?OK, seems reasonable to me.
Quoted 4 lines
Quoted 4 lines
now this is just a 1:1 conversion of the existing code, however for platforms like MTL+ we might need to give more control through
bo_createQuoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
or stop using WB on the CPU side if we need CPU access.
Bspec: 45101, 44235 #xe
Bspec: 70552, 71582, 59400 #xe2Signed-off-by Matthew Auld <[email protected]>Cc Pallavi Mishra <[email protected]>Cc Thomas Hellström <[email protected]>Cc Joonas Lahtinen <[email protected]>Cc Lucas De Marchi <[email protected]>Cc Matt Roper <[email protected]>Cc José Roberto de Souza <[email protected]>Cc Filip Hazubski <[email protected]>Cc Carl Zhang <[email protected]>Cc Effie Yu <[email protected]>---
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/i915/gt/intel_gtt.h | 2 +- drivers/gpu/drm/xe/tests/xe_migrate.c | 2 +- drivers/gpu/drm/xe/xe_ggtt.c | 7 ++- drivers/gpu/drm/xe/xe_ggtt_types.h | 2 +- drivers/gpu/drm/xe/xe_migrate.c | 14 ++--- drivers/gpu/drm/xe/xe_pt.c | 32 +++++------- drivers/gpu/drm/xe/xe_pt.h | 6 +-- drivers/gpu/drm/xe/xe_vm.c | 73+++++++++++++++++++++----
Quoted 6 lines
Quoted 6 lines
-
Quoted 5 lines
Quoted 5 lines
-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm_types.h | 13 +++-- include/uapi/drm/xe_drm.h | 41 ++++++++++++++- 10 files changed, 134 insertions(+), 58 deletions(-)Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Review
Quoted 1 line
(quoted diff) +0
-0
b/drivers/gpu/drm/i915/gt/intel_gtt.hQuoted 5 lines
drivers/gpu/drm/i915/gt/intel_gtt.h struct i915_address_space { · line 302
Quoted 5 lines
index
4d6296cdbcfd..bb4c182048c3100644drivers/gpu/drm/i915/gt/intel_gtt.h +0
-0@@ -302,7 +302,7 @@ struct i915_address_space { (*alloc_scratch_dma)(struct i915_address_space*vm,
Quoted 32 lines
Quoted 32 lines
int
Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Review
Quoted 9 lines
drivers/gpu/drm/i915/gt/intel_gtt.h +1
-1sz);u64 (*pte_encode)(dma_addr_t addr, -unsigned int pat_index, +u32 pat_index,u32 flags);/* Create a valid PTE */ #define PTE_READ_ONLY BIT(0)#define PTE_LM BIT(1)(quoted diff) +0
-0
b/drivers/gpu/drm/xe/tests/xe_migrate.cQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/tests/xe_migrate.c static void xe_migrate_sanity_test(struct
Quoted 4 lines
index
5c8d5e78d9bc..7a128fd20a29100644drivers/gpu/drm/xe/tests/xe_migrate.c +0
-0@@ -301,7 +301,7 @@ static void xe_migrate_sanity_test(struct
xe_migrateQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
*m, struct kunit *test)
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/tests/xe_migrate.c +0
-0/* First part of the test, are we updating our pagetable bo with a new
entry? */
Quoted 1 line
Quoted 1 line
xe_map_wr(xe, &bo->vmap, XE_PAGE_SIZE *(
NUM_KERNEL_PDE-Quoted 10 lines
Quoted 10 lines
1),
Quoted 9 lines
Quoted 9 lines
u64,
Quoted 3 lines
drivers/gpu/drm/xe/tests/xe_migrate.c
Quoted 3 lines
0xdeaddeadbeefbeef);drivers/gpu/drm/xe/tests/xe_migrate.c +1
-1- expected =xe_pte_encode (pt,0 ,XE_CACHE_WB ,0 ); + expected = xe_pte_encode(pt, 0, xe_pat_get_index(xe,
XE_CACHE_WB), 0);Quoted 4 lines
Quoted 4 lines
if (m->q->vm->flags & XE_VM_FLAG_64K) expected |= XE_PTE_PS64; if (xe_bo_is_vram(pt)) diff --git
a/drivers/gpu/drm/xe/xe_ggtt.cQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
b/drivers/gpu/drm/xe/xe_ggtt.cQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c u64 xe_ggtt_pte_encode(struct xe_bo *bo,
Quoted 4 lines
index
209fa053d9fb..4134c26150a5100644drivers/gpu/drm/xe/xe_ggtt.c +0
-0@@ -41,7 +41,8 @@ u64 xe_ggtt_pte_encode(struct xe_bo *bo,u64
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
bo_offset)Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c u64 xe_ggtt_pte_encode(struct xe_bo *bo, · line 44
Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c +0
-1pte |=XE_GGTT_PTE_DM ;if ((ggtt->pat_encode).pte_encode) - pte = (ggtt->pat_encode).pte_encode(xe, pte,
XE_CACHE_WB_1_WAY);Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_ggtt.c u64 xe_ggtt_pte_encode(struct xe_bo *bo, · line 45
Quoted 2 lines
drivers/gpu/drm/xe/xe_ggtt.c +2
-0+ pte = (ggtt->pat_encode).pte_encode(xe, pte, +xe_pat_get_index(xe,
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
XE_CACHE_WB_1_WAY));Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c static void primelockdep(struct xe_ggtt
Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c +0
-0return pte; }@@ -102,10 +103,8 @@ static void primelockdep(struct xe_ggtt*ggtt)
Quoted 88 lines
Quoted 88 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_ggtt.c +0
-0} static u64 xelpg_ggtt_pte_encode_pat(struct xe_device *xe,u64
pte_pat,Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_ggtt.c
Quoted 1 line
drivers/gpu/drm/xe/xe_ggtt.c +0
-1-enum xe_cache_level cache)
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_ggtt.c +1
-2+ u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELPG_GGTT_PTE_PAT_MASK ); if (pat_index & BIT(0)) diff --git
a/drivers/gpu/drm/xe/xe_ggtt_types.h
b/drivers/gpu/drm/xe/xe_ggtt_types.hQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_ggtt_types.h struct xe_ggtt { · line 34
Quoted 8 lines
index
7e55fac1a8a9..0bc40cb072e3100644drivers/gpu/drm/xe/xe_ggtt_types.h +0
-1@@ -31,7 +31,7 @@ struct xe_ggtt {struct { u64 (*pte_encode)(struct xe_device *xe, u64 pte_pat, - enum xe_cache_levelcache);
Quoted 33 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Review
Quoted 5 lines
drivers/gpu/drm/xe/xe_ggtt_types.h +1
-0+ u32 pat_index); } pat_encode; };(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_migrate.cQuoted 16 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
index
a782ea282cb6..54585e98452a100644drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -24,6 +24,7 @@#include "xe_lrc.h" Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_migrate.c +1
-0#include "xe_map.h" #include "xe_mocs.h" +#include "xe_pat.h" #include "xe_pt.h" #include "xe_res_cursor.h" #include "xe_sched_job.h" @@ -162,6 +163,7 @@ static int xe_migrate_prepare_vm(struct
xe_tile*tile,
Quoted 2 lines
Quoted 2 lines
struct
xe_migrate*m,Quoted 1 line
Quoted 1 line
u32
num_entries=NUM_PT_SLOTS,num_level= vm-pt_root[id]-
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
level;
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct
Quoted 8 lines
u32
map_ofs, level, i;
structxe_bo*bo, *batch =tile->mem.kernel_bb_pool->bo;drivers/gpu/drm/xe/xe_migrate.c +1
-0+u32 pat_index =xe_pat_get_index (xe,XE_CACHE_WB );u64 entry;int ret;@@ -189,14 +191,14 @@ static int xe_migrate_prepare_vm(struct
xe_tileQuoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
*tile, struct
xe_migrate*m,Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 192
Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-1return ret; } - entry = xe_pde_encode(bo, bo->size - XE_PAGE_SIZE,
XE_CACHE_WB);Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 194
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ entry = xe_pde_encode(bo, bo->size - XE_PAGE_SIZE,
pat_index);Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 199
Quoted 7 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-1xe_pt_write (xe, &vm->pt_root[id]->bo->vmap,0 , entry); map_ofs = (num_entries - num_level) *XE_PAGE_SIZE ;/* Map the entire BO in our level 0 pt */ for (i =0 , level =0 ; i < num_entries; level++) { - entry = xe_pte_encode(bo, i * XE_PAGE_SIZE,
XE_CACHE_WB,Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
0);
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 201
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ entry = xe_pte_encode(bo, i * XE_PAGE_SIZE,
pat_index, 0);Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 202
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-0xe_map_wr(xe, &bo->vmap, map_ofs + level * 8, u64,entry);
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -214,7 +216,7 @@ static int xe_migrate_prepare_vm(struct
xe_tile*tile,
Quoted 3 lines
Quoted 3 lines
struct
xe_migrate*m,Quoted 2 lines
Quoted 2 lines
for (i = 0; i < batch->size; i += vm->flags & XE_VM_FLAG_64K ?Quoted 7 lines
Quoted 7 lines
XE_64K_PAGE_SIZE:Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 2 lines
XE_PAGE_SIZE) {drivers/gpu/drm/xe/xe_migrate.c +0
-1- entry = xe_pte_encode(batch, i,
XE_CACHE_WB, 0);Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ entry =xe_pte_encode (batch, i, pat_index,0 ); xe_map_wr(xe, &bo->vmap, map_ofs + level*
Quoted 5 lines
Quoted 5 lines
8, u64,
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
entry);drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -239,7 +241,7 @@ static int xe_migrate_prepare_vm(struct
xe_tile*tile,
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
struct
xe_migrate*m,Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 4 lines
flags =
XE_PDE_64K;entry = xe_pde_encode(bo,
map_ofs+ (level - 1) *drivers/gpu/drm/xe/xe_migrate.c +0
-1- XE_PAGE_SIZE,
XE_CACHE_WB);Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ XE_PAGE_SIZE, pat_index); xe_map_wr(xe, &bo->vmap, map_ofs +
XE_PAGE_SIZEQuoted 7 lines
Quoted 7 lines
* level,
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
u64,
Quoted 4 lines
Quoted 4 lines
entry | flags); }drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -247,7 +249,7 @@ static int xe_migrate_prepare_vm(struct
xe_tile*tile,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
struct
xe_migrate*m,Quoted 7 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 7 lines
/* Write PDE's that point to our BO. */ for (i = 0; i < num_entries - num_level; i++) { entry = xe_pde_encode(bo, i * XE_PAGE_SIZE,drivers/gpu/drm/xe/xe_migrate.c +1
-1- XE_CACHE_WB); + pat_index); xe_map_wr(xe, &bo->vmap, map_ofs +
XE_PAGE_SIZEQuoted 132 lines
Quoted 132 lines
+
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(i + 1) * 8, u64, entry); @@ -1256,7 +1258,7@@
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
xe_migrate_update_pgtables(struct
xe_migrateQuoted 5 lines
Quoted 5 lines
*m,
Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 4 lines
XE_WARN_ON(pt_bo->size != SZ_4K);drivers/gpu/drm/xe/xe_migrate.c +0
-1- addr = xe_pte_encode(pt_bo, 0,
XE_CACHE_WB, 0);Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ addr = xe_pte_encode(pt_bo, 0,xe_pat_get_index(xe,
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
XE_CACHE_WB), 0);Quoted 10 lines
Quoted 10 lines
bb->cs[bb->len++] = lower_32_bits(addr); bb->cs[bb->len++] = upper_32_bits(addr); }(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_pt.cindex64713f400d94..019af2920078100644drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -10,6 +10,7 @@#include "xe_gt.h" Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_pt.c +1
-0#include "xe_gt_tlb_invalidation.h" #include "xe_migrate.h" +#include "xe_pat.h" #include "xe_pt_types.h" #include "xe_pt_walk.h" #include "xe_res_cursor.h" @@ -57,24 +58,22 @@ static struct xe_pt *xe_pt_entry(struct
xe_pt_dir
*pt_dir, unsigned int index)Quoted 12 lines
Quoted 12 lines
* * Return: An encoded page directory entry. No errors. */drivers/gpu/drm/xe/xe_pt.c +2
-2-u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level cache) +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 +pat_index) {u64 pde;struct xe_vm *vm = bo->vm;struct xe_device *xe = vm->xe;Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c +1
-2- pde =xe_bo_addr (bo, bo_offset,XE_PAGE_SIZE ); pde |=XE_PAGE_PRESENT |XE_PAGE_RW ;if ((vm->pat_encode).pde_encode) - pde = (vm->pat_encode).pde_encode (xe, pde, cache); + pde = (vm->pat_encode).pde_encode(xe, pde,
pat_index);Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 9 lines
return pde; }drivers/gpu/drm/xe/xe_pt.c +1
-1-static u64 __pte_encode (u64 pte,enum xe_cache_level cache, +static u64 __pte_encode (u64 pte,u32 pat_index,struct xe_vma *vma,u32 pt_level) { struct xe_vm *vm = xe_vma_vm(vma); @@ -89,7 +88,7 @@static
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
u64 __pte_encode(u64 pte, enum
xe_cache_levelcache,Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 5 lines
pte |= XE_PTE_NULL; if ((vm->pat_encode).pte_encode)drivers/gpu/drm/xe/xe_pt.c +1
-1- pte = (vm->pat_encode).pte_encode (xe, pte, cache); + pte = (vm->pat_encode).pte_encode(xe, pte,
pat_index);Quoted 48 lines
Quoted 22 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c static u64 __pte_encode(u64 pte, enum
Quoted 4 lines
if (pt_level == 1) pte |= XE_PDE_PS_2M;drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -112,7 +111,7 @@ static u64 __pte_encode(u64 pte, enum
xe_cache_levelcache,Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c +0
-1* * Return: An encoded page-table entry. No errors. */ -u64 xe_pte_encode(struct xe_bo *bo, u64 offset, enum
xe_cache_levelQuoted 7 lines
Quoted 7 lines
cache,
Quoted 6 lines
drivers/gpu/drm/xe/xe_pt.c u64 xe_pte_encode(struct xe_bo *bo, u64
Quoted 6 lines
drivers/gpu/drm/xe/xe_pt.c +2
-0+u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 +pat_index,u32 pt_level) {u64 pte;@@ -121,7 +120,7 @@ u64 xe_pte_encode(struct xe_bo *bo, u64offset,
Quoted 4 lines
Quoted 4 lines
enum
xe_cache_levelcache,Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c +0
-0if (xe_bo_is_vram (bo) ||xe_bo_is_stolen_devmem (bo)) pte |=XE_PPGTT_PTE_DM ;Quoted 19 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c +1
-1-return __pte_encode (pte, cache,NULL , pt_level); +return __pte_encode (pte, pat_index,NULL , pt_level); } static u64 __xe_pt_empty_pte(struct xe_tile *tile,struct
xe_vm*vm, @@ -134,12 +133,12 @@ static u64 __xe_pt_empty_pte(structxe_tile*tile,
Quoted 11 lines
Quoted 11 lines
struct
xe_vm*vm,Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 5 lines
if (level == 0) { u64 empty = xe_pte_encode(vm->scratch_bo[id], 0,drivers/gpu/drm/xe/xe_pt.c +1
-1- XE_CACHE_WB, 0); + xe_pat_get_index(vm->xe,
XE_CACHE_WB), 0);Quoted 4 lines
Quoted 4 lines
return empty; } else { return xe_pde_encode(vm->scratch_pt[id][level - 1]-bo, 0,
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 2 lines
drivers/gpu/drm/xe/xe_pt.c +1
-1- XE_CACHE_WB); + xe_pat_get_index(vm->xe,
XE_CACHE_WB));Quoted 3 lines
Quoted 3 lines
} }Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_pt.c +0
-2@@ -368,8 +367,6 @@ struct xe_pt_stage_bind_walk {struct xe_vm *vm;/** @tile: The tile we're building for. */ struct xe_tile *tile; -/** @cache: Desired cache level for the ptes */ -enum xe_cache_level cache;/** @default_pte: PTE flag only template. No address is
associated */
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
u64 default_pte; /** @dma_offset: DMA offset to add to the PTE. */ @@-604,7
+601,7 @@ xe_pt_stage_bind_entry(structxe_ptw*parent,Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
pgoff_toffset,Quoted 3 lines
Quoted 3 lines
pte = __pte_encode(is_null ? 0 : xe_res_dma(curs) + xe_walk-
dma_offset,Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +0
-1- xe_walk->cache, xe_walk->vma,level);
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +1
-0+ xe_walk->vma->pat_index,xe_walk-
Quoted 44 lines
Quoted 4 lines
Quoted 4 lines
vma,
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
level);
Quoted 2 lines
Quoted 2 lines
pte |= xe_walk->default_pte;Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind_entry(struct xe_ptw · line 666
Quoted 3 lines
/*drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -669,7 +666,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent,
pgoff_toffset,Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind_entry(struct xe_ptw · line 673
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +0
-1xe_child->is_compact = true; } - pte = xe_pde_encode(xe_child->bo, 0, xe_walk-cache)
Quoted 31 lines
|
Quoted 4 lines
Quoted 4 lines
flags;
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind_entry(struct xe_ptw · line 670
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +1
-0+ pte = xe_pde_encode(xe_child->bo, 0, xe_walk->vma-
pat_index) | flags;Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +0
-0ret = xe_pt_insert_entry(xe_walk, xe_parent, offset,
xe_child,Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +0
-0pte);}drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -730,13 +727,8 @@ xe_pt_stage_bind(struct xe_tile *tile,struct
xe_vmaQuoted 2 lines
Quoted 2 lines
*vma,
Quoted 1 line
Quoted 1 line
if (vma && vma->gpuva.flags &
XE_VMA_ATOMIC_PTE_BIT)Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
xe_walk.default_pte |=
XE_USM_PPGTT_PTE_AE;Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 5 lines
Quoted 1 line
xe_walk.dma_offset = vram_region_gpu_offset(bo-
ttm.resource);
drivers/gpu/drm/xe/xe_pt.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c +0
-3- xe_walk.cache =XE_CACHE_WB ; - } else { - if (!xe_vma_has_no_bo(vma) && bo->flags &
XE_BO_SCANOUT_BIT)Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c +1
-3- xe_walk.cache =XE_CACHE_WT ; - else - xe_walk.cache = XE_CACHE_WB; } +if (!xe_vma_has_no_bo (vma) &&xe_bo_is_stolen (bo)) xe_walk.dma_offset =xe_ttm_stolen_gpu_offset(xe_bo_device(bo));
Quoted 1 line
Quoted 1 line
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_pt.hindex
01be7ab08f87..1d433a5a96b4Quoted 112 lines
Quoted 112 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 5 lines
Quoted 5 lines
100644
drivers/gpu/drm/xe/xe_pt.h +0
-0@@ -45,10 +45,8 @@ __xe_pt_unbind_vma(struct xe_tile *tile,struct
xe_vma*vma, structxe_exec_queuQuoted 11 lines
Quoted 11 lines
bool xe_pt_zap_ptes(struct
xe_tile*tile, structxe_vma
*vma);drivers/gpu/drm/xe/xe_pt.h +2
-3-u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level level); +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 +pat_index); -u64 xe_pte_encode(struct xe_bo *bo, u64 offset, enum
xe_cache_levelcache,
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.h
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.h +3
-1- u32 pt_level); +u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 +pat_index, +u32
pt_level);Quoted 1 line
Quoted 1 line
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
#endif(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_vm.cindex
7eeeed0411f3..34603a7e84b0Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
100644
drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -6,6 +6,7 @@#include "xe_vm.h" Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +1
-0#include <linux/dma-fence-array.h> +#include <linux/nospec.h> #include <drm/drm_print.h> #include <drm/ttm/ttm_execbuf_util.h> @@ -874,7 +875,8@@ static struct
xe_vma*xe_vma_create(struct
xe_vmQuoted 1 line
Quoted 1 line
*vm,
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c static struct xe_vma
Quoted 10 lines
u64 start, u64 end, bool read_only, bool is_null,drivers/gpu/drm/xe/xe_vm.c +2
-1- u8 tile_mask) + u8 tile_mask, + u32 pat_index) {struct xe_vma *vma;struct xe_tile *tile;@@ -913,6 +915,8 @@ static struct xe_vma*xe_vma_create(struct
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
xe_vmQuoted 10 lines
Quoted 10 lines
*vm,
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c static void
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c +2
-0vma->tile_mask |=0x1 << id; } + vma->pat_index = pat_index; +if (vm->xe->info.platform ==XE_PVC ) vma->gpuva.flags |=XE_VMA_ATOMIC_PTE_BIT ;@@ -1194,10 +1198,8 @@ static voidxe_vma_op_work_func(struct
Quoted 28 lines
Quoted 28 lines
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
Quoted 7 lines
work_struct*w);Quoted 6 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0static void vm_destroy_work_func (struct work_struct *w); static u64 xelp_ppgtt_pde_encode_pat(struct xe_device *xe,u64
pde_pat,Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-1-enum xe_cache_level cache)
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c +1
-2+ u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pde_pat &= ~(XELP_PDE_PAT_MASK ); if (pat_index & BIT(0)) @@ -1213,10 +1215,8 @@ staticu64
xelp_ppgtt_pde_encode_pat(struct
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
xe_device*xe, u64pde_pat,Quoted 5 lines
Quoted 5 lines
}
static u64 xelp_ppgtt_pte_encode_pat(struct
xe_device
*xe,
u64
pte_pat,Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-1-enum xe_cache_level cache)
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c +1
-2+ u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELP_PTE_PAT_MASK ); if (pat_index & BIT(0)) @@ -1622,7 +1622,7 @@ struct
xe_vm*xe_vm_lookup(struct
xe_fileQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
*xef,
Quoted 3 lines
Quoted 3 lines
u32 id)
Quoted 2 lines
Quoted 2 lines
u64 xe_vm_pdp4_descriptor(struct
xe_vm*vm, structxe_tile*tile)
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 4 lines
{ return xe_pde_encode(vm->pt_root[tile->id]->bo, 0,drivers/gpu/drm/xe/xe_vm.c +1
-1- XE_CACHE_WB); + xe_pat_get_index(vm->xe,
XE_CACHE_WB));Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
} static struct dma_fence * @@ -2311,7 +2311,7 @@ staticvoid print_op(struct
xe_device*xe,struct
Quoted 3 lines
Quoted 3 lines
drm_gpuva_op*op)Quoted 2 lines
Quoted 2 lines
static struct
drm_gpuva_ops* vm_bind_ioctl_ops_create(structxe_vm*vm, structxe_bo*bo,
Quoted 267 lines
Quoted 267 lines
Quoted 57 lines
Quoted 57 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
u64 bo_offset_or_userptr, u64 addr, u64range,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- u32 operation, u8 tile_mask, u32 region) + u32 operation, u8 tile_mask, u32 region, u32
pat_index)Quoted 4 lines
Quoted 4 lines
{ struct drm_gem_object *obj = bo ? &bo->ttm.base : NULL; struct ww_acquire_ctx ww; @@ -2339,6 +2339,7 @@vm_bind_ioctl_ops_create(struct
xe_vm*vm,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
struct
xe_bo*bo,Quoted 1 line
Quoted 1 line
struct
xe_vma_op*op =
gpuva_op_to_vma_op(__op);Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 5 lines
op->tile_mask = tile_mask;drivers/gpu/drm/xe/xe_vm.c +1
-0+ op->pat_index = pat_index; op->map.immediate = operation &
XE_VM_BIND_FLAG_IMMEDIATE;Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c vm_bind_ioctl_ops_create(struct xe_vm
Quoted 2 lines
op->map.read_only =drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2366,6 +2367,7 @@ vm_bind_ioctl_ops_create(struct xe_vm*vm,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
struct
xe_bo*bo,Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0struct xe_vma_op *op =
gpuva_op_to_vma_op(__op);Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c vm_bind_ioctl_ops_create(struct xe_vm
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c +1
-0op->tile_mask = tile_mask; + op->pat_index = pat_index; op->prefetch.region = region; }break ;@@ -2408,7 +2410,8 @@ vm_bind_ioctl_ops_create(struct xe_vm*vm,
Quoted 24 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
struct
xe_bo*bo,Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c vm_bind_ioctl_ops_create(struct xe_vm · line 2412
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0} static struct xe_vma *new_vma(struct xe_vm *vm, struct
drm_gpuva_op_map*op,Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c vm_bind_ioctl_ops_create(struct xe_vm · line 2416
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c +2
-1- u8 tile_mask, bool read_only, bool is_null) + u8 tile_mask, bool read_only, bool is_null, + u32 pat_index) { struct xe_bo *bo = op->gem.obj ? gem_to_xe_bo(op-gem.obj) :
Quoted 13 lines
NULL;
Quoted 12 lines
Quoted 12 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0struct xe_vma *vma;@@ -2425,7 +2428,7 @@ static struct xe_vma *new_vma(struct
xe_vm*vm, struct
drm_gpuva_op_map*op,Quoted 8 lines
Quoted 8 lines
vma = xe_vma_create(vm, bo, op->gem.offset, op->va.addr, op->va.addr + op->va.range - 1, read_only, is_null,drivers/gpu/drm/xe/xe_vm.c +1
-1- tile_mask); + tile_mask, pat_index);if (bo)xe_bo_unlock (bo, &ww);Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c static int
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2539,7 +2542,7 @@ static intvm_bind_ioctl_ops_parse(struct
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
xe_vmQuoted 3 lines
Quoted 3 lines
*vm, struct
xe_exec_queue*q,Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0vma = new_vma(vm, &op-base.map,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0op->tile_mask, op-Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
map.read_only,
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2544
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- op->map.is_null); + op->map.is_null, op-
pat_index);Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c static int
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2567,7 +2570,7 @@ static intvm_bind_ioctl_ops_parse(struct
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
xe_vmQuoted 5 lines
Quoted 5 lines
*vm, struct
xe_exec_queue*q,Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2570
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0vma = new_vma(vm, op-base.remap.prev,
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0op->tile_mask,Quoted 8 lines
Quoted 8 lines
read_only,Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2572
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- is_null); + is_null, op-
pat_index);Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c static int
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2603,7 +2606,7 @@ static intvm_bind_ioctl_ops_parse(struct
Quoted 22 lines
Quoted 22 lines
Quoted 6 lines
Quoted 6 lines
xe_vmQuoted 5 lines
Quoted 5 lines
*vm, struct
xe_exec_queue*q,Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2606
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0vma = new_vma(vm, op-base.remap.next,
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0op->tile_mask,
read_only,Quoted 15 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2608
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- is_null); + is_null, op-
pat_index);Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -3158,8 +3161,14 @@ static intvm_bind_ioctl_check_args(struct
xe_device*xe,Quoted 6 lines
Quoted 6 lines
u32 obj = (*bind_ops)[i].obj; u64 obj_offset = (*bind_ops)[i].obj_offset; u32 region = (*bind_ops)[i].region;drivers/gpu/drm/xe/xe_vm.c +1
-0+u32 pat_index = (*bind_ops)[i].pat_index;bool is_null = op &XE_VM_BIND_FLAG_NULL ;Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+ if (XE_IOCTL_DBG(xe, pat_index >= xe-info.pat_table_n_entries)) {
drivers/gpu/drm/xe/xe_vm.c
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +4
-0+ err = -EINVAL ; + goto free_bind_ops; + } +if (i ==0 ) { *async = !!(op &
XE_VM_BIND_FLAG_ASYNC);Quoted 40 lines
Quoted 40 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
} else if (XE_IOCTL_DBG(xe, !*async) || @@ -3346,8+3355,25
Quoted 15 lines
Quoted 14 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
@@ int xe_vm_bind_ioctl(struct
drm_device*dev,
Quoted 12 lines
Quoted 12 lines
void *data, struct
drm_file*file)Quoted 11 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 11 lines
struct drm_gem_object *gem_obj; u64 range = bind_ops[i].range; u64 addr = bind_ops[i].addr;drivers/gpu/drm/xe/xe_vm.c +6
-0+u32 op = bind_ops[i].op;u32 obj = bind_ops[i].obj;u64 obj_offset = bind_ops[i].obj_offset; +u32 pat_index = bind_ops[i].pat_index; +u16 coh_mode; + + pat_index = array_index_nospec(pat_index, + xe-info.pat_table_n_entries);
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+ coh_mode = xe_pat_index_get_coh_mode(xe,
pat_index);Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+if (XE_IOCTL_DBG (xe, !coh_mode)) {Assuming we drop the unusable entries from the TGL table, this should be impossible, right? Any index that makes it past the
n_entriescheck at the top of the function should have a valid, non-zero
coh_modevalue.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
So this should probably be an assertion (to highlight a KMD bug) rather than just a silent uapi failure return.
Makes sense.
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c +5
-0+ err = -EINVAL ; + goto put_obj; + } + + if (XE_IOCTL_DBG(xe, VM_BIND_OP(op) ==
XE_VM_BIND_OP_MAP_USERPTR&&Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+ coh_mode ==
XE_GEM_COHERENCY_NONE))Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
{
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c +3
-0+ err = -EINVAL ; + goto put_obj; + }if (!obj)continue ;@@ -3375,6 +3401,22 @@ int xe_vm_bind_ioctl(struct
drm_deviceQuoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
*dev,
Quoted 13 lines
Quoted 13 lines
void *data, struct
drm_file*file)Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3406
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +3
-0goto put_obj; } } + +if (bos[i]->coh_mode) { + if (XE_IOCTL_DBG(xe, bos[i]->coh_mode !=
coh_mode)) {Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3410
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +4
-0+ err = -EINVAL ; + goto put_obj; + } + } else if (XE_IOCTL_DBG(xe, coh_mode ==
XE_GEM_COHERENCY_NONE)) {Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3412
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +2
-0+/* +* Imported dma-buf from a different device should
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3413
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+* require 1way or 2way coherency since we don't
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
know
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3414
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+* how it was mapped on CPU. Just assume is it
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c +5
-0+* potentially cached on CPU side. +*/ + err = -EINVAL ; + goto put_obj; + } } if (args->num_syncs) { @@ -3412,10 +3454,11 @@ intxe_vm_bind_ioctl(struct
drm_deviceQuoted 39 lines
Quoted 39 lines
Quoted 8 lines
Quoted 8 lines
*dev,
Quoted 7 lines
Quoted 7 lines
void *data, struct
drm_file*file)Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 6 lines
u64 obj_offset = bind_ops[i].obj_offset; u8 tile_mask = bind_ops[i].tile_mask; u32 region = bind_ops[i].region;drivers/gpu/drm/xe/xe_vm.c +1
-0+u32 pat_index = bind_ops[i].pat_index; ops[i] = vm_bind_ioctl_ops_create(vm, bos[i],
obj_offset,Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
addr, range, op,
tile_mask,Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
Review
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- region); + region, pat_index);if (IS_ERR (ops[i])) { err =PTR_ERR (ops[i]); ops[i] =NULL ;(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_vm_types.hQuoted 7 lines
Quoted 7 lines
index
83a1f87b6537..508679ed3c74100644drivers/gpu/drm/xe/xe_vm_types.h +0
-0@@ -111,6 +111,11 @@ struct xe_vma {*/ u8 tile_present;Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vma { · line 115
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm_types.h +2
-0+/** +* @pat_index: The pat index to use when encoding the PTEs for
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vma { · line 116
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +1
-0+thisvma.
Quoted 9 lines
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm_types.h +3
-0+*/ +u32 pat_index; +struct {struct list_head rebind_link; } notifier;@@ -338,10 +343,8 @@ struct xe_vm {bool batch_invalidate_tlb;Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 342
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm_types.h +0
-2struct { -u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat, - enum xe_cache_levelcache);
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 343
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +0
-1- u64 (*pde_encode)(struct xe_device *xe, u64
pde_pat,Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 344
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +0
-1-enum xe_cache_level cache);
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 346
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +1
-0+ u64 (*pte_encode)(struct xe_device *xe, u64 pte_pat,u32
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
pat_index);Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 347
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +1
-0+ u64 (*pde_encode)(struct xe_device *xe, u64
pde_pat,Quoted 14 lines
Quoted 14 lines
u32
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
pat_index);Quoted 12 lines
Quoted 12 lines
drivers/gpu/drm/xe/xe_vm_types.h +2
-0} pat_encode; };@@ -417,6 +420,8 @@ struct xe_vma_op {struct async_op_fence *fence;/** @tile_mask: gt mask for this operation */ u8 tile_mask; +/** @pat_index: The pat index to use for this operation. */ +u32 pat_index;/** @flags: operation flags */ enum xe_vma_op_flags flags;Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(quoted diff) +0
-0
b/include/uapi/drm/xe_drm.hindex
64bc66d4b550..0c15b6f32447Quoted 117 lines
Quoted 117 lines
Quoted 24 lines
Quoted 24 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 604
Quoted 11 lines
100644
include/uapi/drm/xe_drm.h +2
-2@@ -600,8 +600,45 @@ struct drm_xe_vm_bind_op {*/ __u32 obj; -/** @pad: MBZ */ -__u32 pad; +/** +* @pat_index: The platform defined @pat_index to use for this
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
mapping.
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 605
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* The index basically maps to some predefined memory attributes,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 607
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * including things like caching, coherency and likely +other stuffin
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 609
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * the future. The exact meaning of the pat_index is +platform specificBTW, "other stuff in the future" already includes compression on Xe2, we just haven't landed the patches for the Xe2 table yet.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Ok, good to know.
Quoted 2 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 610
Quoted 2 lines
include/uapi/drm/xe_drm.h +1
-0+* and defined in the Bspec and PRMs. When the KMD sets up
Quoted 88 lines
Quoted 88 lines
the
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
binding
Quoted 4 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 614
Quoted 4 lines
include/uapi/drm/xe_drm.h +4
-0+* the index here is encoded into the ppGTT PTE. +* +* For coherency the @pat_index needs to match the +* drm_xe_gem_create.coh_mode, so either
XE_GEM_COHERENCY_NONE,Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 615
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* XE_GEM_COHERENCY_1WAY or
XE_GEM_COHERENCY_2WAY. TheQuoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
KMD will extract
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 616
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* the coherency mode from the @pat_index and reject if there is a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 619
Quoted 3 lines
include/uapi/drm/xe_drm.h +3
-0+* mismatch (see note below for pre-MTL platforms). +* +* Note: On pre-MTL platforms there is only a caching mode and no
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 620
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* explicit coherency mode, but on such hardware there is always a
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 622
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * shared-LLC (or is dgpu) so all GT memory accesses are +coherentwith
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 623
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* CPU caches even with the caching mode set as uncached. It's
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 624
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+onlythe
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 625
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* display engine that is incoherent (on dgpu it must be in VRAM which
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 626
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* is always mapped as WC on the CPU). However to keep the uapi
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
somewhat
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 627
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* consistent with newer platforms the KMD groups the different
Quoted 14 lines
Quoted 14 lines
cache
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 629
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * levels into the following coherency buckets on all +pre-MTL platforms:
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 634
Quoted 5 lines
include/uapi/drm/xe_drm.h +5
-0+* +* ppGTT UC -> XE_GEM_COHERENCY_NONE +* ppGTT WC -> XE_GEM_COHERENCY_NONE +* ppGTT WT -> XE_GEM_COHERENCY_NONE +* ppGTT WB -> XE_GEM_COHERENCY_2WAY As noted on the previous patch, it seems like 2-way is appropriate for LLC platforms, but 1-way might be a more accurate description of dGPU behavior.
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 637
Quoted 3 lines
include/uapi/drm/xe_drm.h +3
-0+* +* In practice UC/WC/WT should only ever used for scanout +surfaces on
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 639
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * such platforms since it is only the display engine that +isactually
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 6 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 641
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * incoherent. Everything else should typically use WB +given that weWhat if we're sharing our buffers with some other (non-GPU) device? Are there cases where that other device wouldn't be coherent with the LLC, so we'd want to use one of these?
Yeah, I guess there might be cases like that. I'll reword.
Quoted 2 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Matt
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 643
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * have a shared-LLC. On MTL+ this completely changes +(alsopotentially
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 644
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* no shared-LLC) and the HW defines the coherency mode as part
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
+of
the
Quoted 3 lines
include/uapi/drm/xe_drm.h
Quoted 3 lines
include/uapi/drm/xe_drm.h +3
-0+* @pat_index. +* +* Note: For userptr and externally imported dma-buf the kernel
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
expects
Quoted 10 lines
Quoted 9 lines
include/uapi/drm/xe_drm.h +3
-1+* either 1WAY or 2WAY for the @pat_index. +*/ +__u32 pat_index;union {/** -- 2.41.0
Re: [Intel-xe] [RFC 5/5] drm/xe/uapi: support pat_index selection with vm_bind
Quoted 224 lines · Carl
Quoted 216 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Tuesday, September 5, 2023 5:47 PMQuoted 127 lines · Carl
Quoted 127 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Monday, September 4, 2023 5:24 PMQuoted 9 lines · Carl
Quoted 9 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Thursday, August 31, 2023 6:44 PMQuoted 2 lines · Carl
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Thursday, August 31, 2023 12:02 AMQuoted 2 lines · Carl
Several questions:
1. thepat_indexfromvm_bindwill override the setting from
bo_create?Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
How to keep the value from
bo_createunchanged?You only set the
pat_indexduringvm_bind. Atgem_createyou just need tell the kernel what CPU side caching mode to use for system memory (wb/wc/uc), plus the expected GPU coherency mode of thepat_index. The current expectation is that the coherency mode of thepat_indexshould match the one atgem_create.Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
It means that we set the incomplete
pat_indexatgem_create(coherency
only) And set another part atvm_bind(gpu cache) Why we separate them? Why not set a full value atvm_bind?You always give the full
pat_indexatvm_bind. It's platform specific but the BSpec for the most part tells you what the coherency mode of the
pat_indexis.Quoted 4 lines
Quoted 4 lines
For example if you look at the MTL
pat_indextable it tells you thecoh_mode:Quoted 8 lines
Quoted 8 lines
pat_index0-2 =coh_mode"No snoop" =COH_NONEpat_index 3 = coh_mode "1way" = COH_1WAY pat_index 4 = coh_mode "2way" = COH_2WAYOn future platforms the table is larger and also encodes stuff like compression, but KMD really only cares about the
coh_mode, since that will restrict the allowed CPUsmem_cachingvalues that userspace canselect.
Quoted 4 lines
Quoted 4 lines
From KMD pov this matters since we always give userspace zeroed
memory, which is useful for normal applications but is also needed forsecurity reasons.
Quoted 11 lines
Quoted 11 lines
However the actual clearing might result in the writes only being visible in the CPU cache (not flushed to main memory), and if the GPU doesn't snoop the CPU cache it can read directly from main memory, which effectively bypasses the clearing. To prevent that the KMD needs to know how userspace will map the memory via the GPU i.e what is the
coh_modefor thepat_index. The gist is that if you selectCOH_NONEthen it needs to clflush the pages before giving them to userspace, which is done by forcing you to use uc/wc for thesmem_caching(the kernel will internally issue cflush on x86 when marking the pages as uc/wc). And if you select wb there is no flushing, buthere you need to use 1way or 2way.
Quoted 5 lines
Quoted 1 line
It sounds like a perf optimization, if no considering the perf,
you could always call Clflush to flush the cpu cache to main memory before giving it to user spaceI think the other thing is that the KMD is no longer doing any manual flushing, but instead the core kernel will now do that for us when marking the pages as wc/uc on x86.
Quoted 6 lines
Quoted 6 lines
AFAIK separating them is just to prevent userspace from mixing different coherency modes for the same object by having it immutable (you can still use different
pat_indexbut the coherency must becompatible).
Quoted 6 lines
Quoted 6 lines
Also at creation time we can determine if the
smem_cachingis compatible with thecoh_mode.Note that it might be that we ditch the
COH_2WAYand just haveCOH_NONEandCOH_AT_LEAST_1WAY, in which case this might be slightlydifferent.
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 4 lines
Quoted 4 lines
2. no UC/WB/WC definition (CPU cachable) in
drm_xe_gem_mmap_offset,Quoted 9 lines
will it be handled by KMD automatically?
Quoted 3 lines
For example: if set 1-way, it means GPU could snoop CPU cache, we could use WB in mmap offset If it is
COHERENCY_NONE, we could only useUC, all these logic is handled by KMD automatically?
Yes, the mmap will use the same CPU side caching mode that you set at
gem_createwithsmem_caching(wb/wc/uc).It means it could not be changed after
gem_create. Sometimes, UMD doesn’t know Whether the data need to be accessed by CPU atgem_create. only application Knows the purpose of allocation & the surface usage. So, may need change UMD api to let applications set whether theQuoted 7 lines
Quoted 7 lines
resource is "accessible", or it will hurt the perf.
Right, the
smem_cachingis immutable.Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
3. about " For imported dma-buf (from a different device) the coherency mode is also implicit and must also be either 1WAY or
2WAY"
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
it means it must be 1way or 2way, and UMD need not to set it?
If you are going to
vm_bindit you need to supply thepat_indexand in this case it needs to be either 1way or 2way. The object comes from a different device so we might not have smem_caching/coh_mode like we do for native objects.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
How about the GPU cache setting of imported dma-buf?
Is there possible conflict between dma-buf producer and consumer? For example:
In producer side , one bo is set GPU L3 cache, also CPU cacheable, it should be 2Way In client side, we know the coherency should be 1-way or 2-way, if it suppose no GPU cache Used. It just set 1-way.
Quoted 43 lines
Quoted 43 lines
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
It may cause some problems?
If the producer and consumer are the same device, then I think the original gem_create.coh_mode is considered, since it's effectively still a native object. In which case the normal rules apply.
Quoted 17 lines
But the consumer does not know the
coh_mode, so , it is possible that it set will set Another value, then it will failed. And because consumer does not know it is 1-way or 2-way, consumer could not Set the gpu cache correctly. Because if it is 1-way, cpu could not snoop gpu cacheQuoted 11 lines
If it comes from a different device then we don't know what gem_create.coh_mode is (or if that even exists) since we don't have the usual native object, but just some opaque dma-buf object. In this case it must be at least 1way or 2way.
From KMD pov we only care about userspace not being able to
bypass the clearing. There might still be other coherency issues though.Quoted 2 lines
Also , when a bo is created, UMD does not know whether it will be
exported.
Quoted 6 lines
Quoted 6 lines
Quoted 1 line
It maybe be set as
COHERENCY_NONE, right?Yeah, you can use whatever you want. Just that your
smem_cachingwill need to be wc/uc if usingCOH_NONE. On the importer side, if it comes from the same device then it's just a normal native object and the normalrules apply.
Quoted 5 lines
Quoted 5 lines
Currently this would mean also using
pat_indexwithCOH_NONE. If it is imported from a different device then you need to use apat_indexwith 1way or 2way, since KMD doesn't really know how the object is mapped on the CPU or if the CPU caches are dirty (potentially withclearing).
Quoted 6 lines
Quoted 3 lines
Is this going to be problematic?
If imported surfaces is from same device, how consumer know it is original
coh_nonOr 1-way or 2-way?Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
And it also means producer and consumer could have different
pat_indexforsame bo.
Ok, if KMD instead allows 1way or 2way for dma-buf when it's external or from the same device? Would that help here? i.e you can always select 1way or 2way, even if it came from the same device and was
coh_none?Quoted 3 lines
Quoted 3 lines
Other option is you could call the
vm_bindioctl to check if thecoh_modeis supported by checking if it returns an error? But maybe that istoo nasy.
Quoted 4 lines
Quoted 1 line
Still a bit confuse, I create a bo with
coh_none, then export it to a PRIME fd. In consumer side, it does not know it iscoh_none, and will callvm_bindtoset
pat_index.Quoted 2 lines
You mean it will return a error when I set different coh value though
vm_bind?Quoted 1 line
And try
coh_non, 1-way, 2-way, then get the correct one? TBH, it is ugly.Yeah, it is nasty.
Quoted 2 lines
My concern is: if it is
coh_nonor 1-way, it means I could not use GPU cache ifthe bo also is CPU accessible.
Quoted 4 lines
Because CPU could not snoop GPU cache.
So, GPU cache specified bypat_indexalso will be rejected?Could we add some query interface to query the CPU cache and coherency
setting of a bo?
Quoted 8 lines
Say if the buffer comes from a different device + driver, what
pat_indexwould you choose for that? There is no such thing as coh_mode/smem_caching for such an object. If you have to handle that case can't you just treat all imported dma-buf the same as that? The new proposal was to allow 1way or 2way for any dma-buf object, even if it comes from the same device and has different BOcoh_mode. Do you need more than that? AFAICT the query would only really work for objects imported from the same device.
Quoted 3 lines
Looks I misunderstood the proposal, so, you mean if it shared in one device, Consumer could choose 1-way ,2-way , any one should work , and no error returned, no rejection from
vm_bind, right?
Yeah, treat same-device and different-device the same in KMD, and allow 1way or 2way (anything but coh_none), since you highlighted the issue that for importer from same device it might not even know the original coh_mode.
Quoted 1,029 lines
Quoted 1,029 lines
Quoted 60 lines
Quoted 60 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Quoted 11 lines
-----Original Message-----
From: Auld, Matthew <[email protected]>
Sent: Wednesday, August 30, 2023 7:28 PM
To: Roper, Matthew D <[email protected]>
Cc: [email protected]; Mishra, PallaviQuoted 3 lines
Quoted 3 lines
Thomas Hellström <[email protected]>; Joonas Lahtinen <[email protected]>; De Marchi, Lucas <[email protected]>; Souza, Jose <[email protected]>;
Hazubski,
Quoted 2 lines
Quoted 2 lines
Filip <[email protected]>; Zhang, Carl
<[email protected]>; Yu,Effie
Quoted 3 lines
Quoted 3 lines
<[email protected]>
Subject: Re: [RFC 5/5] drm/xe/uapi: supportpat_indexselection withQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
vm_bindQuoted 7 lines
Quoted 7 lines · Matt Roper
Quoted 5 lines · Matthew Auld
Quoted 4 lines
Allow userspace to directly control the
pat_indexfor a given vm binding. This should allow directly controlling the coherency, caching and potentially other stuff in the future for theppGTT binding.
Quoted 29 lines
Quoted 29 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
The exact meaning behind the
pat_indexis very platform specific (see BSpec or PRMs) but effectively maps to some predefined memory attributes. From the KMD pov we only care about the coherency that is provided by thepat_index, which falls into eitherNONE, 1WAY or 2WAY.
Quoted 22 lines
Quoted 22 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The
vm_bindcoherency mode for the givenpat_indexneeds to matchthe
Quoted 19 lines
Quoted 19 lines
Quoted 10 lines
Quoted 2 lines
given
coh_modethat was set at object creation. For platforms that lackIs it actually important to match the coherency mode? I think one of the main goals was to know up front if userspace might be using a non-snooping PAT setting that would let it bypass the CPU cache (and potentially read old, stale data from a different process if the buffer's clear value is still sitting in cache and hasn't landed in memory yet).
Quoted 6 lines
If that's the only concern, then I think it should still be fine to map with a non-matching PAT as long as it's more coherent than the one specified at creation, right? E.g., if the buffer was created with 1-way coherency, it would be fine to map it with 2-way because userspace still can't use that to observe the previous contents of the buffer. Or
Yeah, I guess we could in theory do something that.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
if the buffer was created with "non-coherent" then we've already done the necessary clflushing in kernel before handing to buffer over to userspace to ensure the clear value landed in memory, so any valid PAT index should be safe (from a security POV) after that, right? Any other problems that arise from mismatched coherency would just be contained to the app possibly shooting itself in the foot, which isn't really our
concern.
Quoted 450 lines
Quoted 450 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
That is also my understanding, at least from the KMD security pov. If you allocate as wb then you must use at least 1way, since there is no flushing
for
Quoted 2 lines
Quoted 2 lines
clearing or swap-in. For uc/wc you could in theory use whatever you
want.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
the explicit coherency mode, we treat UC/WT/WC as NONE and
WB
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
as
2WAY.
Quoted 3 lines
Quoted 3 lines
For userptr mappings we lack a corresponding gem object, so the
expected
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
coherency mode is instead implicit and must fall into either 1WAY or 2WAY. Trying to use NONE will be rejected by the kernel. For imported dma-buf (from a different device) the coherency mode is also implicit and must also be either 1WAY or
2WAY.
Quoted 38 lines
Quoted 38 lines
Quoted 38 lines
Quoted 38 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
As part of adding
pat_indexsupport withvm_bindwe also need stopusing
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
xe_cache_leveland instead use thepat_indexin various places. We still make use ofxe_cache_level, but only as a convenience for kernel internal objectsi (internally it maps to some reasonablepat_index). ForQuoted 13 lines
Quoted 4 lines
Quoted 4 lines
Maybe we should kill
xe_cache_levelcompletely and just assignxe_gt->pat_cached/xe_gt->pat_uncachedat init that can be used in appropriate places, similar to what we do with MOCS (gt-mocs.uc_index,
Quoted 8 lines
Quoted 1 line
gt->mocs.wb_index)?OK, seems reasonable to me.
Quoted 4 lines
Quoted 4 lines
now this is just a 1:1 conversion of the existing code, however for platforms like MTL+ we might need to give more control through
bo_createQuoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
or stop using WB on the CPU side if we need CPU access.
Bspec: 45101, 44235 #xe
Bspec: 70552, 71582, 59400 #xe2Signed-off-by Matthew Auld <[email protected]>Cc Pallavi Mishra <[email protected]>Cc Thomas Hellström <[email protected]>Cc Joonas Lahtinen <[email protected]>Cc Lucas De Marchi <[email protected]>Cc Matt Roper <[email protected]>Cc José Roberto de Souza <[email protected]>Cc Filip Hazubski <[email protected]>Cc Carl Zhang <[email protected]>Cc Effie Yu <[email protected]>---
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/i915/gt/intel_gtt.h | 2 +- drivers/gpu/drm/xe/tests/xe_migrate.c | 2 +- drivers/gpu/drm/xe/xe_ggtt.c | 7 ++- drivers/gpu/drm/xe/xe_ggtt_types.h | 2 +- drivers/gpu/drm/xe/xe_migrate.c | 14 ++--- drivers/gpu/drm/xe/xe_pt.c | 32 +++++------- drivers/gpu/drm/xe/xe_pt.h | 6 +-- drivers/gpu/drm/xe/xe_vm.c | 73+++++++++++++++++++++----
Quoted 6 lines
Quoted 6 lines
-
Quoted 5 lines
Quoted 5 lines
-
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm_types.h | 13 +++-- include/uapi/drm/xe_drm.h | 41 ++++++++++++++- 10 files changed, 134 insertions(+), 58 deletions(-)Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Review
Quoted 1 line
(quoted diff) +0
-0
b/drivers/gpu/drm/i915/gt/intel_gtt.hQuoted 5 lines
drivers/gpu/drm/i915/gt/intel_gtt.h struct i915_address_space { · line 302
Quoted 5 lines
index
4d6296cdbcfd..bb4c182048c3100644drivers/gpu/drm/i915/gt/intel_gtt.h +0
-0@@ -302,7 +302,7 @@ struct i915_address_space { (*alloc_scratch_dma)(struct i915_address_space*vm,
Quoted 32 lines
Quoted 32 lines
int
Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Review
Quoted 9 lines
drivers/gpu/drm/i915/gt/intel_gtt.h +1
-1sz);u64 (*pte_encode)(dma_addr_t addr, -unsigned int pat_index, +u32 pat_index,u32 flags);/* Create a valid PTE */ #define PTE_READ_ONLY BIT(0)#define PTE_LM BIT(1)(quoted diff) +0
-0
b/drivers/gpu/drm/xe/tests/xe_migrate.cQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/tests/xe_migrate.c static void xe_migrate_sanity_test(struct
Quoted 4 lines
index
5c8d5e78d9bc..7a128fd20a29100644drivers/gpu/drm/xe/tests/xe_migrate.c +0
-0@@ -301,7 +301,7 @@ static void xe_migrate_sanity_test(struct
xe_migrateQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
*m, struct kunit *test)
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/tests/xe_migrate.c +0
-0/* First part of the test, are we updating our pagetable bo with a new
entry? */
Quoted 1 line
Quoted 1 line
xe_map_wr(xe, &bo->vmap, XE_PAGE_SIZE *(
NUM_KERNEL_PDE-Quoted 10 lines
Quoted 10 lines
1),
Quoted 9 lines
Quoted 9 lines
u64,
Quoted 3 lines
drivers/gpu/drm/xe/tests/xe_migrate.c
Quoted 3 lines
0xdeaddeadbeefbeef);drivers/gpu/drm/xe/tests/xe_migrate.c +1
-1- expected =xe_pte_encode (pt,0 ,XE_CACHE_WB ,0 ); + expected = xe_pte_encode(pt, 0, xe_pat_get_index(xe,
XE_CACHE_WB), 0);Quoted 4 lines
Quoted 4 lines
if (m->q->vm->flags & XE_VM_FLAG_64K) expected |= XE_PTE_PS64; if (xe_bo_is_vram(pt)) diff --git
a/drivers/gpu/drm/xe/xe_ggtt.cQuoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
b/drivers/gpu/drm/xe/xe_ggtt.cQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c u64 xe_ggtt_pte_encode(struct xe_bo *bo,
Quoted 4 lines
index
209fa053d9fb..4134c26150a5100644drivers/gpu/drm/xe/xe_ggtt.c +0
-0@@ -41,7 +41,8 @@ u64 xe_ggtt_pte_encode(struct xe_bo *bo,u64
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
bo_offset)Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c u64 xe_ggtt_pte_encode(struct xe_bo *bo, · line 44
Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c +0
-1pte |=XE_GGTT_PTE_DM ;if ((ggtt->pat_encode).pte_encode) - pte = (ggtt->pat_encode).pte_encode(xe, pte,
XE_CACHE_WB_1_WAY);Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_ggtt.c u64 xe_ggtt_pte_encode(struct xe_bo *bo, · line 45
Quoted 2 lines
drivers/gpu/drm/xe/xe_ggtt.c +2
-0+ pte = (ggtt->pat_encode).pte_encode(xe, pte, +xe_pat_get_index(xe,
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
XE_CACHE_WB_1_WAY));Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c static void primelockdep(struct xe_ggtt
Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c +0
-0return pte; }@@ -102,10 +103,8 @@ static void primelockdep(struct xe_ggtt*ggtt)
Quoted 88 lines
Quoted 88 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_ggtt.c +0
-0} static u64 xelpg_ggtt_pte_encode_pat(struct xe_device *xe,u64
pte_pat,Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_ggtt.c
Quoted 1 line
drivers/gpu/drm/xe/xe_ggtt.c +0
-1-enum xe_cache_level cache)
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_ggtt.c +1
-2+ u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELPG_GGTT_PTE_PAT_MASK ); if (pat_index & BIT(0)) diff --git
a/drivers/gpu/drm/xe/xe_ggtt_types.h
b/drivers/gpu/drm/xe/xe_ggtt_types.hQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_ggtt_types.h struct xe_ggtt { · line 34
Quoted 8 lines
index
7e55fac1a8a9..0bc40cb072e3100644drivers/gpu/drm/xe/xe_ggtt_types.h +0
-1@@ -31,7 +31,7 @@ struct xe_ggtt {struct { u64 (*pte_encode)(struct xe_device *xe, u64 pte_pat, - enum xe_cache_levelcache);
Quoted 33 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Review
Quoted 5 lines
drivers/gpu/drm/xe/xe_ggtt_types.h +1
-0+ u32 pat_index); } pat_encode; };(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_migrate.cQuoted 16 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
index
a782ea282cb6..54585e98452a100644drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -24,6 +24,7 @@#include "xe_lrc.h" Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_migrate.c +1
-0#include "xe_map.h" #include "xe_mocs.h" +#include "xe_pat.h" #include "xe_pt.h" #include "xe_res_cursor.h" #include "xe_sched_job.h" @@ -162,6 +163,7 @@ static int xe_migrate_prepare_vm(struct
xe_tile*tile,
Quoted 2 lines
Quoted 2 lines
struct
xe_migrate*m,Quoted 1 line
Quoted 1 line
u32
num_entries=NUM_PT_SLOTS,num_level= vm-pt_root[id]-
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
level;
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct
Quoted 8 lines
u32
map_ofs, level, i;
structxe_bo*bo, *batch =tile->mem.kernel_bb_pool->bo;drivers/gpu/drm/xe/xe_migrate.c +1
-0+u32 pat_index =xe_pat_get_index (xe,XE_CACHE_WB );u64 entry;int ret;@@ -189,14 +191,14 @@ static int xe_migrate_prepare_vm(struct
xe_tileQuoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
*tile, struct
xe_migrate*m,Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 192
Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-1return ret; } - entry = xe_pde_encode(bo, bo->size - XE_PAGE_SIZE,
XE_CACHE_WB);Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 194
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ entry = xe_pde_encode(bo, bo->size - XE_PAGE_SIZE,
pat_index);Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 199
Quoted 7 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-1xe_pt_write (xe, &vm->pt_root[id]->bo->vmap,0 , entry); map_ofs = (num_entries - num_level) *XE_PAGE_SIZE ;/* Map the entire BO in our level 0 pt */ for (i =0 , level =0 ; i < num_entries; level++) { - entry = xe_pte_encode(bo, i * XE_PAGE_SIZE,
XE_CACHE_WB,Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
0);
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 201
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ entry = xe_pte_encode(bo, i * XE_PAGE_SIZE,
pat_index, 0);Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_prepare_vm(struct · line 202
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-0xe_map_wr(xe, &bo->vmap, map_ofs + level * 8, u64,entry);
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -214,7 +216,7 @@ static int xe_migrate_prepare_vm(struct
xe_tile*tile,
Quoted 3 lines
Quoted 3 lines
struct
xe_migrate*m,Quoted 2 lines
Quoted 2 lines
for (i = 0; i < batch->size; i += vm->flags & XE_VM_FLAG_64K ?Quoted 7 lines
Quoted 7 lines
XE_64K_PAGE_SIZE:Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 2 lines
XE_PAGE_SIZE) {drivers/gpu/drm/xe/xe_migrate.c +0
-1- entry = xe_pte_encode(batch, i,
XE_CACHE_WB, 0);Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ entry =xe_pte_encode (batch, i, pat_index,0 ); xe_map_wr(xe, &bo->vmap, map_ofs + level*
Quoted 5 lines
Quoted 5 lines
8, u64,
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
entry);drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -239,7 +241,7 @@ static int xe_migrate_prepare_vm(struct
xe_tile*tile,
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
struct
xe_migrate*m,Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 4 lines
flags =
XE_PDE_64K;entry = xe_pde_encode(bo,
map_ofs+ (level - 1) *drivers/gpu/drm/xe/xe_migrate.c +0
-1- XE_PAGE_SIZE,
XE_CACHE_WB);Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ XE_PAGE_SIZE, pat_index); xe_map_wr(xe, &bo->vmap, map_ofs +
XE_PAGE_SIZEQuoted 7 lines
Quoted 7 lines
* level,
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
u64,
Quoted 4 lines
Quoted 4 lines
entry | flags); }drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -247,7 +249,7 @@ static int xe_migrate_prepare_vm(struct
xe_tile*tile,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
struct
xe_migrate*m,Quoted 7 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 7 lines
/* Write PDE's that point to our BO. */ for (i = 0; i < num_entries - num_level; i++) { entry = xe_pde_encode(bo, i * XE_PAGE_SIZE,drivers/gpu/drm/xe/xe_migrate.c +1
-1- XE_CACHE_WB); + pat_index); xe_map_wr(xe, &bo->vmap, map_ofs +
XE_PAGE_SIZEQuoted 132 lines
Quoted 132 lines
+
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
(i + 1) * 8, u64, entry); @@ -1256,7 +1258,7@@
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
xe_migrate_update_pgtables(struct
xe_migrateQuoted 5 lines
Quoted 5 lines
*m,
Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c
Quoted 4 lines
XE_WARN_ON(pt_bo->size != SZ_4K);drivers/gpu/drm/xe/xe_migrate.c +0
-1- addr = xe_pte_encode(pt_bo, 0,
XE_CACHE_WB, 0);Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ addr = xe_pte_encode(pt_bo, 0,xe_pat_get_index(xe,
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
XE_CACHE_WB), 0);Quoted 10 lines
Quoted 10 lines
bb->cs[bb->len++] = lower_32_bits(addr); bb->cs[bb->len++] = upper_32_bits(addr); }(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_pt.cindex64713f400d94..019af2920078100644drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -10,6 +10,7 @@#include "xe_gt.h" Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_pt.c +1
-0#include "xe_gt_tlb_invalidation.h" #include "xe_migrate.h" +#include "xe_pat.h" #include "xe_pt_types.h" #include "xe_pt_walk.h" #include "xe_res_cursor.h" @@ -57,24 +58,22 @@ static struct xe_pt *xe_pt_entry(struct
xe_pt_dir
*pt_dir, unsigned int index)Quoted 12 lines
Quoted 12 lines
* * Return: An encoded page directory entry. No errors. */drivers/gpu/drm/xe/xe_pt.c +2
-2-u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level cache) +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 +pat_index) {u64 pde;struct xe_vm *vm = bo->vm;struct xe_device *xe = vm->xe;Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c +1
-2- pde =xe_bo_addr (bo, bo_offset,XE_PAGE_SIZE ); pde |=XE_PAGE_PRESENT |XE_PAGE_RW ;if ((vm->pat_encode).pde_encode) - pde = (vm->pat_encode).pde_encode (xe, pde, cache); + pde = (vm->pat_encode).pde_encode(xe, pde,
pat_index);Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 9 lines
return pde; }drivers/gpu/drm/xe/xe_pt.c +1
-1-static u64 __pte_encode (u64 pte,enum xe_cache_level cache, +static u64 __pte_encode (u64 pte,u32 pat_index,struct xe_vma *vma,u32 pt_level) { struct xe_vm *vm = xe_vma_vm(vma); @@ -89,7 +88,7 @@static
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
Quoted 1 line
u64 __pte_encode(u64 pte, enum
xe_cache_levelcache,Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 5 lines
pte |= XE_PTE_NULL; if ((vm->pat_encode).pte_encode)drivers/gpu/drm/xe/xe_pt.c +1
-1- pte = (vm->pat_encode).pte_encode (xe, pte, cache); + pte = (vm->pat_encode).pte_encode(xe, pte,
pat_index);Quoted 48 lines
Quoted 22 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c static u64 __pte_encode(u64 pte, enum
Quoted 4 lines
if (pt_level == 1) pte |= XE_PDE_PS_2M;drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -112,7 +111,7 @@ static u64 __pte_encode(u64 pte, enum
xe_cache_levelcache,Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c +0
-1* * Return: An encoded page-table entry. No errors. */ -u64 xe_pte_encode(struct xe_bo *bo, u64 offset, enum
xe_cache_levelQuoted 7 lines
Quoted 7 lines
cache,
Quoted 6 lines
drivers/gpu/drm/xe/xe_pt.c u64 xe_pte_encode(struct xe_bo *bo, u64
Quoted 6 lines
drivers/gpu/drm/xe/xe_pt.c +2
-0+u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 +pat_index,u32 pt_level) {u64 pte;@@ -121,7 +120,7 @@ u64 xe_pte_encode(struct xe_bo *bo, u64offset,
Quoted 4 lines
Quoted 4 lines
enum
xe_cache_levelcache,Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c +0
-0if (xe_bo_is_vram (bo) ||xe_bo_is_stolen_devmem (bo)) pte |=XE_PPGTT_PTE_DM ;Quoted 19 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c +1
-1-return __pte_encode (pte, cache,NULL , pt_level); +return __pte_encode (pte, pat_index,NULL , pt_level); } static u64 __xe_pt_empty_pte(struct xe_tile *tile,struct
xe_vm*vm, @@ -134,12 +133,12 @@ static u64 __xe_pt_empty_pte(structxe_tile*tile,
Quoted 11 lines
Quoted 11 lines
struct
xe_vm*vm,Quoted 5 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 5 lines
if (level == 0) { u64 empty = xe_pte_encode(vm->scratch_bo[id], 0,drivers/gpu/drm/xe/xe_pt.c +1
-1- XE_CACHE_WB, 0); + xe_pat_get_index(vm->xe,
XE_CACHE_WB), 0);Quoted 4 lines
Quoted 4 lines
return empty; } else { return xe_pde_encode(vm->scratch_pt[id][level - 1]-bo, 0,
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 2 lines
drivers/gpu/drm/xe/xe_pt.c +1
-1- XE_CACHE_WB); + xe_pat_get_index(vm->xe,
XE_CACHE_WB));Quoted 3 lines
Quoted 3 lines
} }Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_pt.c +0
-2@@ -368,8 +367,6 @@ struct xe_pt_stage_bind_walk {struct xe_vm *vm;/** @tile: The tile we're building for. */ struct xe_tile *tile; -/** @cache: Desired cache level for the ptes */ -enum xe_cache_level cache;/** @default_pte: PTE flag only template. No address is
associated */
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
u64 default_pte; /** @dma_offset: DMA offset to add to the PTE. */ @@-604,7
+601,7 @@ xe_pt_stage_bind_entry(structxe_ptw*parent,Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
pgoff_toffset,Quoted 3 lines
Quoted 3 lines
pte = __pte_encode(is_null ? 0 : xe_res_dma(curs) + xe_walk-
dma_offset,Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +0
-1- xe_walk->cache, xe_walk->vma,level);
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +1
-0+ xe_walk->vma->pat_index,xe_walk-
Quoted 44 lines
Quoted 4 lines
Quoted 4 lines
vma,
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
level);
Quoted 2 lines
Quoted 2 lines
pte |= xe_walk->default_pte;Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind_entry(struct xe_ptw · line 666
Quoted 3 lines
/*drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -669,7 +666,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent,
pgoff_toffset,Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind_entry(struct xe_ptw · line 673
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +0
-1xe_child->is_compact = true; } - pte = xe_pde_encode(xe_child->bo, 0, xe_walk-cache)
Quoted 31 lines
|
Quoted 4 lines
Quoted 4 lines
flags;
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c xe_pt_stage_bind_entry(struct xe_ptw · line 670
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +1
-0+ pte = xe_pde_encode(xe_child->bo, 0, xe_walk->vma-
pat_index) | flags;Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +0
-0ret = xe_pt_insert_entry(xe_walk, xe_parent, offset,
xe_child,Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +0
-0pte);}drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -730,13 +727,8 @@ xe_pt_stage_bind(struct xe_tile *tile,struct
xe_vmaQuoted 2 lines
Quoted 2 lines
*vma,
Quoted 1 line
Quoted 1 line
if (vma && vma->gpuva.flags &
XE_VMA_ATOMIC_PTE_BIT)Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
xe_walk.default_pte |=
XE_USM_PPGTT_PTE_AE;Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 5 lines
Quoted 1 line
xe_walk.dma_offset = vram_region_gpu_offset(bo-
ttm.resource);
drivers/gpu/drm/xe/xe_pt.c
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c +0
-3- xe_walk.cache =XE_CACHE_WB ; - } else { - if (!xe_vma_has_no_bo(vma) && bo->flags &
XE_BO_SCANOUT_BIT)Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c
Quoted 7 lines
drivers/gpu/drm/xe/xe_pt.c +1
-3- xe_walk.cache =XE_CACHE_WT ; - else - xe_walk.cache = XE_CACHE_WB; } +if (!xe_vma_has_no_bo (vma) &&xe_bo_is_stolen (bo)) xe_walk.dma_offset =xe_ttm_stolen_gpu_offset(xe_bo_device(bo));
Quoted 1 line
Quoted 1 line
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_pt.hindex
01be7ab08f87..1d433a5a96b4Quoted 112 lines
Quoted 112 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 5 lines
Quoted 5 lines
100644
drivers/gpu/drm/xe/xe_pt.h +0
-0@@ -45,10 +45,8 @@ __xe_pt_unbind_vma(struct xe_tile *tile,struct
xe_vma*vma, structxe_exec_queuQuoted 11 lines
Quoted 11 lines
bool xe_pt_zap_ptes(struct
xe_tile*tile, structxe_vma
*vma);drivers/gpu/drm/xe/xe_pt.h +2
-3-u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level level); +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 +pat_index); -u64 xe_pte_encode(struct xe_bo *bo, u64 offset, enum
xe_cache_levelcache,
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.h
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.h +3
-1- u32 pt_level); +u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 +pat_index, +u32
pt_level);Quoted 1 line
Quoted 1 line
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
#endif(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_vm.cindex
7eeeed0411f3..34603a7e84b0Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
100644
drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -6,6 +6,7 @@#include "xe_vm.h" Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +1
-0#include <linux/dma-fence-array.h> +#include <linux/nospec.h> #include <drm/drm_print.h> #include <drm/ttm/ttm_execbuf_util.h> @@ -874,7 +875,8@@ static struct
xe_vma*xe_vma_create(struct
xe_vmQuoted 1 line
Quoted 1 line
*vm,
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c static struct xe_vma
Quoted 10 lines
u64 start, u64 end, bool read_only, bool is_null,drivers/gpu/drm/xe/xe_vm.c +2
-1- u8 tile_mask) + u8 tile_mask, + u32 pat_index) {struct xe_vma *vma;struct xe_tile *tile;@@ -913,6 +915,8 @@ static struct xe_vma*xe_vma_create(struct
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
xe_vmQuoted 10 lines
Quoted 10 lines
*vm,
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c static void
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c +2
-0vma->tile_mask |=0x1 << id; } + vma->pat_index = pat_index; +if (vm->xe->info.platform ==XE_PVC ) vma->gpuva.flags |=XE_VMA_ATOMIC_PTE_BIT ;@@ -1194,10 +1198,8 @@ static voidxe_vma_op_work_func(struct
Quoted 28 lines
Quoted 28 lines
Quoted 9 lines
Quoted 9 lines
Quoted 7 lines
Quoted 7 lines
work_struct*w);Quoted 6 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0static void vm_destroy_work_func (struct work_struct *w); static u64 xelp_ppgtt_pde_encode_pat(struct xe_device *xe,u64
pde_pat,Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-1-enum xe_cache_level cache)
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c +1
-2+ u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pde_pat &= ~(XELP_PDE_PAT_MASK ); if (pat_index & BIT(0)) @@ -1213,10 +1215,8 @@ staticu64
xelp_ppgtt_pde_encode_pat(struct
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
xe_device*xe, u64pde_pat,Quoted 5 lines
Quoted 5 lines
}
static u64 xelp_ppgtt_pte_encode_pat(struct
xe_device
*xe,
u64
pte_pat,Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-1-enum xe_cache_level cache)
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c +1
-2+ u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELP_PTE_PAT_MASK ); if (pat_index & BIT(0)) @@ -1622,7 +1622,7 @@ struct
xe_vm*xe_vm_lookup(struct
xe_fileQuoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
*xef,
Quoted 3 lines
Quoted 3 lines
u32 id)
Quoted 2 lines
Quoted 2 lines
u64 xe_vm_pdp4_descriptor(struct
xe_vm*vm, structxe_tile*tile)
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 4 lines
{ return xe_pde_encode(vm->pt_root[tile->id]->bo, 0,drivers/gpu/drm/xe/xe_vm.c +1
-1- XE_CACHE_WB); + xe_pat_get_index(vm->xe,
XE_CACHE_WB));Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
} static struct dma_fence * @@ -2311,7 +2311,7 @@ staticvoid print_op(struct
xe_device*xe,struct
Quoted 3 lines
Quoted 3 lines
drm_gpuva_op*op)Quoted 2 lines
Quoted 2 lines
static struct
drm_gpuva_ops* vm_bind_ioctl_ops_create(structxe_vm*vm, structxe_bo*bo,
Quoted 267 lines
Quoted 267 lines
Quoted 57 lines
Quoted 57 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
u64 bo_offset_or_userptr, u64 addr, u64range,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- u32 operation, u8 tile_mask, u32 region) + u32 operation, u8 tile_mask, u32 region, u32
pat_index)Quoted 4 lines
Quoted 4 lines
{ struct drm_gem_object *obj = bo ? &bo->ttm.base : NULL; struct ww_acquire_ctx ww; @@ -2339,6 +2339,7 @@vm_bind_ioctl_ops_create(struct
xe_vm*vm,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
struct
xe_bo*bo,Quoted 1 line
Quoted 1 line
struct
xe_vma_op*op =
gpuva_op_to_vma_op(__op);Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 5 lines
op->tile_mask = tile_mask;drivers/gpu/drm/xe/xe_vm.c +1
-0+ op->pat_index = pat_index; op->map.immediate = operation &
XE_VM_BIND_FLAG_IMMEDIATE;Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c vm_bind_ioctl_ops_create(struct xe_vm
Quoted 2 lines
op->map.read_only =drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2366,6 +2367,7 @@ vm_bind_ioctl_ops_create(struct xe_vm*vm,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
struct
xe_bo*bo,Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0struct xe_vma_op *op =
gpuva_op_to_vma_op(__op);Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c vm_bind_ioctl_ops_create(struct xe_vm
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c +1
-0op->tile_mask = tile_mask; + op->pat_index = pat_index; op->prefetch.region = region; }break ;@@ -2408,7 +2410,8 @@ vm_bind_ioctl_ops_create(struct xe_vm*vm,
Quoted 24 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
struct
xe_bo*bo,Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c vm_bind_ioctl_ops_create(struct xe_vm · line 2412
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0} static struct xe_vma *new_vma(struct xe_vm *vm, struct
drm_gpuva_op_map*op,Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c vm_bind_ioctl_ops_create(struct xe_vm · line 2416
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c +2
-1- u8 tile_mask, bool read_only, bool is_null) + u8 tile_mask, bool read_only, bool is_null, + u32 pat_index) { struct xe_bo *bo = op->gem.obj ? gem_to_xe_bo(op-gem.obj) :
Quoted 13 lines
NULL;
Quoted 12 lines
Quoted 12 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0struct xe_vma *vma;@@ -2425,7 +2428,7 @@ static struct xe_vma *new_vma(struct
xe_vm*vm, struct
drm_gpuva_op_map*op,Quoted 8 lines
Quoted 8 lines
vma = xe_vma_create(vm, bo, op->gem.offset, op->va.addr, op->va.addr + op->va.range - 1, read_only, is_null,drivers/gpu/drm/xe/xe_vm.c +1
-1- tile_mask); + tile_mask, pat_index);if (bo)xe_bo_unlock (bo, &ww);Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c static int
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -2539,7 +2542,7 @@ static intvm_bind_ioctl_ops_parse(struct
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
xe_vmQuoted 3 lines
Quoted 3 lines
*vm, struct
xe_exec_queue*q,Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0vma = new_vma(vm, &op-base.map,
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0op->tile_mask, op-Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
map.read_only,
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2544
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- op->map.is_null); + op->map.is_null, op-
pat_index);Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c static int
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2567,7 +2570,7 @@ static intvm_bind_ioctl_ops_parse(struct
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
xe_vmQuoted 5 lines
Quoted 5 lines
*vm, struct
xe_exec_queue*q,Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2570
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0vma = new_vma(vm, op-base.remap.prev,
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0op->tile_mask,Quoted 8 lines
Quoted 8 lines
read_only,Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2572
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- is_null); + is_null, op-
pat_index);Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c static int
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2603,7 +2606,7 @@ static intvm_bind_ioctl_ops_parse(struct
Quoted 22 lines
Quoted 22 lines
Quoted 6 lines
Quoted 6 lines
xe_vmQuoted 5 lines
Quoted 5 lines
*vm, struct
xe_exec_queue*q,Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2606
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0vma = new_vma(vm, op-base.remap.next,
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +0
-0op->tile_mask,
read_only,Quoted 15 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c static int · line 2608
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- is_null); + is_null, op-
pat_index);Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 5 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -3158,8 +3161,14 @@ static intvm_bind_ioctl_check_args(struct
xe_device*xe,Quoted 6 lines
Quoted 6 lines
u32 obj = (*bind_ops)[i].obj; u64 obj_offset = (*bind_ops)[i].obj_offset; u32 region = (*bind_ops)[i].region;drivers/gpu/drm/xe/xe_vm.c +1
-0+u32 pat_index = (*bind_ops)[i].pat_index;bool is_null = op &XE_VM_BIND_FLAG_NULL ;Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+ if (XE_IOCTL_DBG(xe, pat_index >= xe-info.pat_table_n_entries)) {
drivers/gpu/drm/xe/xe_vm.c
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +4
-0+ err = -EINVAL ; + goto free_bind_ops; + } +if (i ==0 ) { *async = !!(op &
XE_VM_BIND_FLAG_ASYNC);Quoted 40 lines
Quoted 40 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
} else if (XE_IOCTL_DBG(xe, !*async) || @@ -3346,8+3355,25
Quoted 15 lines
Quoted 14 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
@@ int xe_vm_bind_ioctl(struct
drm_device*dev,
Quoted 12 lines
Quoted 12 lines
void *data, struct
drm_file*file)Quoted 11 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 11 lines
struct drm_gem_object *gem_obj; u64 range = bind_ops[i].range; u64 addr = bind_ops[i].addr;drivers/gpu/drm/xe/xe_vm.c +6
-0+u32 op = bind_ops[i].op;u32 obj = bind_ops[i].obj;u64 obj_offset = bind_ops[i].obj_offset; +u32 pat_index = bind_ops[i].pat_index; +u16 coh_mode; + + pat_index = array_index_nospec(pat_index, + xe-info.pat_table_n_entries);
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+ coh_mode = xe_pat_index_get_coh_mode(xe,
pat_index);Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+if (XE_IOCTL_DBG (xe, !coh_mode)) {Assuming we drop the unusable entries from the TGL table, this should be impossible, right? Any index that makes it past the
n_entriescheck at the top of the function should have a valid, non-zero
coh_modevalue.Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
So this should probably be an assertion (to highlight a KMD bug) rather than just a silent uapi failure return.
Makes sense.
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c +5
-0+ err = -EINVAL ; + goto put_obj; + } + + if (XE_IOCTL_DBG(xe, VM_BIND_OP(op) ==
XE_VM_BIND_OP_MAP_USERPTR&&Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+ coh_mode ==
XE_GEM_COHERENCY_NONE))Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
{
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm.c +3
-0+ err = -EINVAL ; + goto put_obj; + }if (!obj)continue ;@@ -3375,6 +3401,22 @@ int xe_vm_bind_ioctl(struct
drm_deviceQuoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
*dev,
Quoted 13 lines
Quoted 13 lines
void *data, struct
drm_file*file)Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3406
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +3
-0goto put_obj; } } + +if (bos[i]->coh_mode) { + if (XE_IOCTL_DBG(xe, bos[i]->coh_mode !=
coh_mode)) {Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3410
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +4
-0+ err = -EINVAL ; + goto put_obj; + } + } else if (XE_IOCTL_DBG(xe, coh_mode ==
XE_GEM_COHERENCY_NONE)) {Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3412
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +2
-0+/* +* Imported dma-buf from a different device should
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3413
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+* require 1way or 2way coherency since we don't
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
know
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct · line 3414
Quoted 1 line
drivers/gpu/drm/xe/xe_vm.c +1
-0+* how it was mapped on CPU. Just assume is it
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm.c +5
-0+* potentially cached on CPU side. +*/ + err = -EINVAL ; + goto put_obj; + } } if (args->num_syncs) { @@ -3412,10 +3454,11 @@ intxe_vm_bind_ioctl(struct
drm_deviceQuoted 39 lines
Quoted 39 lines
Quoted 8 lines
Quoted 8 lines
*dev,
Quoted 7 lines
Quoted 7 lines
void *data, struct
drm_file*file)Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c
Quoted 6 lines
u64 obj_offset = bind_ops[i].obj_offset; u8 tile_mask = bind_ops[i].tile_mask; u32 region = bind_ops[i].region;drivers/gpu/drm/xe/xe_vm.c +1
-0+u32 pat_index = bind_ops[i].pat_index; ops[i] = vm_bind_ioctl_ops_create(vm, bos[i],
obj_offset,Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
addr, range, op,
tile_mask,Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
Review
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1- region); + region, pat_index);if (IS_ERR (ops[i])) { err =PTR_ERR (ops[i]); ops[i] =NULL ;(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_vm_types.hQuoted 7 lines
Quoted 7 lines
index
83a1f87b6537..508679ed3c74100644drivers/gpu/drm/xe/xe_vm_types.h +0
-0@@ -111,6 +111,11 @@ struct xe_vma {*/ u8 tile_present;Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vma { · line 115
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm_types.h +2
-0+/** +* @pat_index: The pat index to use when encoding the PTEs for
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vma { · line 116
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +1
-0+thisvma.
Quoted 9 lines
Quoted 9 lines
drivers/gpu/drm/xe/xe_vm_types.h +3
-0+*/ +u32 pat_index; +struct {struct list_head rebind_link; } notifier;@@ -338,10 +343,8 @@ struct xe_vm {bool batch_invalidate_tlb;Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 342
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm_types.h +0
-2struct { -u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat, - enum xe_cache_levelcache);
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 343
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +0
-1- u64 (*pde_encode)(struct xe_device *xe, u64
pde_pat,Quoted 6 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 344
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +0
-1-enum xe_cache_level cache);
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 346
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +1
-0+ u64 (*pte_encode)(struct xe_device *xe, u64 pte_pat,u32
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
pat_index);Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vm { · line 347
Quoted 1 line
drivers/gpu/drm/xe/xe_vm_types.h +1
-0+ u64 (*pde_encode)(struct xe_device *xe, u64
pde_pat,Quoted 14 lines
Quoted 14 lines
u32
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
pat_index);Quoted 12 lines
Quoted 12 lines
drivers/gpu/drm/xe/xe_vm_types.h +2
-0} pat_encode; };@@ -417,6 +420,8 @@ struct xe_vma_op {struct async_op_fence *fence;/** @tile_mask: gt mask for this operation */ u8 tile_mask; +/** @pat_index: The pat index to use for this operation. */ +u32 pat_index;/** @flags: operation flags */ enum xe_vma_op_flags flags;Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
(quoted diff) +0
-0
b/include/uapi/drm/xe_drm.hindex
64bc66d4b550..0c15b6f32447Quoted 117 lines
Quoted 117 lines
Quoted 24 lines
Quoted 24 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 604
Quoted 11 lines
100644
include/uapi/drm/xe_drm.h +2
-2@@ -600,8 +600,45 @@ struct drm_xe_vm_bind_op {*/ __u32 obj; -/** @pad: MBZ */ -__u32 pad; +/** +* @pat_index: The platform defined @pat_index to use for this
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
mapping.
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 605
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* The index basically maps to some predefined memory attributes,
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 607
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * including things like caching, coherency and likely +other stuffin
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 609
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * the future. The exact meaning of the pat_index is +platform specificBTW, "other stuff in the future" already includes compression on Xe2, we just haven't landed the patches for the Xe2 table yet.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Ok, good to know.
Quoted 2 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 610
Quoted 2 lines
include/uapi/drm/xe_drm.h +1
-0+* and defined in the Bspec and PRMs. When the KMD sets up
Quoted 88 lines
Quoted 88 lines
the
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
binding
Quoted 4 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 614
Quoted 4 lines
include/uapi/drm/xe_drm.h +4
-0+* the index here is encoded into the ppGTT PTE. +* +* For coherency the @pat_index needs to match the +* drm_xe_gem_create.coh_mode, so either
XE_GEM_COHERENCY_NONE,Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 615
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* XE_GEM_COHERENCY_1WAY or
XE_GEM_COHERENCY_2WAY. TheQuoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
KMD will extract
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 616
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* the coherency mode from the @pat_index and reject if there is a
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 619
Quoted 3 lines
include/uapi/drm/xe_drm.h +3
-0+* mismatch (see note below for pre-MTL platforms). +* +* Note: On pre-MTL platforms there is only a caching mode and no
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 620
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* explicit coherency mode, but on such hardware there is always a
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 622
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * shared-LLC (or is dgpu) so all GT memory accesses are +coherentwith
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 623
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* CPU caches even with the caching mode set as uncached. It's
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 624
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+onlythe
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 625
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* display engine that is incoherent (on dgpu it must be in VRAM which
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 626
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* is always mapped as WC on the CPU). However to keep the uapi
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
somewhat
Quoted 1 line
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 627
Quoted 1 line
include/uapi/drm/xe_drm.h +1
-0+* consistent with newer platforms the KMD groups the different
Quoted 14 lines
Quoted 14 lines
cache
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
include/uapi/drm/xe_drm.h struct drm_xe_vm_bind_op { · line 629
Quoted 2 lines
include/uapi/drm/xe_drm.h +2
-0+ * levels into the following coherency buckets on all +pre-MTL platforms:
10 lines not shown · raw message
Quoted 6 lines
6 lines not shown · raw message
36 lines not shown · raw message
Part of this message is not shown here. View the raw message.
Re: [Intel-xe] [RFC 5/5] drm/xe/uapi: support pat_index selection with vm_bind
In reply to Matthew AuldQuoted 487 lines · Matthew Auld
Allow userspace to directly control the
pat_indexfor a given vm binding. This should allow directly controlling the coherency, caching and potentially other stuff in the future for the ppGTT binding.The exact meaning behind the
pat_indexis very platform specific (see BSpec or PRMs) but effectively maps to some predefined memory attributes. From the KMD pov we only care about the coherency that is provided by thepat_index, which falls into either NONE, 1WAY or 2WAY. Thevm_bindcoherency mode for the givenpat_indexneeds to match the givencoh_modethat was set at object creation. For platforms that lack the explicit coherency mode, we treat UC/WT/WC as NONE and WB as 2WAY.For userptr mappings we lack a corresponding gem object, so the expected coherency mode is instead implicit and must fall into either 1WAY or 2WAY. Trying to use NONE will be rejected by the kernel. For imported dma-buf (from a different device) the coherency mode is also implicit and must also be either 1WAY or 2WAY.
As part of adding
pat_indexsupport withvm_bindwe also need stop usingxe_cache_leveland instead use thepat_indexin various places. We still make use ofxe_cache_level, but only as a convenience for kernel internal objectsi (internally it maps to some reasonablepat_index). For now this is just a 1:1 conversion of the existing code, however for platforms like MTL+ we might need to give more control throughbo_createor stop using WB on the CPU side if we need CPU access.Bspec: 45101, 44235 #xe
Bspec: 70552, 71582, 59400 #xe2Signed-off-by Matthew Auld <[email protected]>Cc Pallavi Mishra <[email protected]>Cc Thomas Hellström <[email protected]>Cc Joonas Lahtinen <[email protected]>Cc Lucas De Marchi <[email protected]>Cc Matt Roper <[email protected]>Cc José Roberto de Souza <[email protected]>Cc Filip Hazubski <[email protected]>Cc Carl Zhang <[email protected]>Cc Effie Yu <[email protected]>--- drivers/gpu/drm/i915/gt/intel_gtt.h | 2 +- drivers/gpu/drm/xe/tests/xe_migrate.c | 2 +- drivers/gpu/drm/xe/xe_ggtt.c | 7 ++- drivers/gpu/drm/xe/xe_ggtt_types.h | 2 +- drivers/gpu/drm/xe/xe_migrate.c | 14 ++--- drivers/gpu/drm/xe/xe_pt.c | 32 +++++------- drivers/gpu/drm/xe/xe_pt.h | 6 +-- drivers/gpu/drm/xe/xe_vm.c | 73 +++++++++++++++++++++------ drivers/gpu/drm/xe/xe_vm_types.h | 13 +++-- include/uapi/drm/xe_drm.h | 41 ++++++++++++++- 10 files changed, 134 insertions(+), 58 deletions(-)drivers/gpu/drm/i915/gt/intel_gtt.h +1
-1@@ -302,7 +302,7 @@ struct i915_address_space { (*alloc_scratch_dma)(struct i915_address_space *vm, int sz);u64 (*pte_encode)(dma_addr_t addr, -unsigned int pat_index, +u32 pat_index,u32 flags);/* Create a valid PTE */ #define PTE_READ_ONLY BIT(0)#define PTE_LM BIT(1)drivers/gpu/drm/xe/tests/xe_migrate.c +1
-1@@ -301,7 +301,7 @@ static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test)/* First part of the test, are we updating our pagetable bo with a new entry? */ xe_map_wr (xe, &bo->vmap,XE_PAGE_SIZE * (NUM_KERNEL_PDE -1 ), u64,0xdeaddeadbeefbeef ); - expected =xe_pte_encode (pt,0 ,XE_CACHE_WB ,0 ); + expected =xe_pte_encode (pt,0 ,xe_pat_get_index (xe,XE_CACHE_WB ),0 );if (m->q->vm->flags &XE_VM_FLAG_64K ) expected |=XE_PTE_PS64 ;if (xe_bo_is_vram (pt))drivers/gpu/drm/xe/xe_ggtt.c +3
-4@@ -41,7 +41,8 @@ u64 xe_ggtt_pte_encode(struct xe_bo *bo, u64 bo_offset) pte |=XE_GGTT_PTE_DM ;if ((ggtt->pat_encode).pte_encode) - pte = (ggtt->pat_encode).pte_encode (xe, pte,XE_CACHE_WB_1_WAY ); + pte = (ggtt->pat_encode).pte_encode (xe, pte, +xe_pat_get_index (xe,XE_CACHE_WB_1_WAY ));return pte; }@@ -102,10 +103,8 @@ static void primelockdep(struct xe_ggtt *ggtt) }static u64 xelpg_ggtt_pte_encode_pat (struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELPG_GGTT_PTE_PAT_MASK );if (pat_index &BIT (0 ))drivers/gpu/drm/xe/xe_ggtt_types.h +1
-1@@ -31,7 +31,7 @@ struct xe_ggtt {struct {u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache); +u32 pat_index); } pat_encode; };drivers/gpu/drm/xe/xe_migrate.c +8
-6@@ -24,6 +24,7 @@#include "xe_lrc.h" #include "xe_map.h" #include "xe_mocs.h" +#include "xe_pat.h" #include "xe_pt.h" #include "xe_res_cursor.h" #include "xe_sched_job.h" @@ -162,6 +163,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,u32 num_entries =NUM_PT_SLOTS , num_level = vm->pt_root[id]->level;u32 map_ofs, level, i;struct xe_bo *bo, *batch = tile->mem.kernel_bb_pool->bo; +u32 pat_index =xe_pat_get_index (xe,XE_CACHE_WB );u64 entry;int ret;@@ -189,14 +191,14 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,return ret; } - entry =xe_pde_encode (bo, bo->size -XE_PAGE_SIZE ,XE_CACHE_WB ); + entry =xe_pde_encode (bo, bo->size -XE_PAGE_SIZE , pat_index);xe_pt_write (xe, &vm->pt_root[id]->bo->vmap,0 , entry); map_ofs = (num_entries - num_level) *XE_PAGE_SIZE ;/* Map the entire BO in our level 0 pt */ for (i =0 , level =0 ; i < num_entries; level++) { - entry =xe_pte_encode (bo, i *XE_PAGE_SIZE ,XE_CACHE_WB ,0 ); + entry =xe_pte_encode (bo, i *XE_PAGE_SIZE , pat_index,0 );xe_map_wr (xe, &bo->vmap, map_ofs + level *8 , u64, entry);@@ -214,7 +216,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,for (i =0 ; i < batch->size; i += vm->flags &XE_VM_FLAG_64K ?XE_64K_PAGE_SIZE :XE_PAGE_SIZE ) { - entry =xe_pte_encode (batch, i,XE_CACHE_WB ,0 ); + entry =xe_pte_encode (batch, i, pat_index,0 );xe_map_wr (xe, &bo->vmap, map_ofs + level *8 , u64, entry);@@ -239,7 +241,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m, flags =XE_PDE_64K ; entry =xe_pde_encode (bo, map_ofs + (level -1 ) * -XE_PAGE_SIZE ,XE_CACHE_WB ); +XE_PAGE_SIZE , pat_index);xe_map_wr (xe, &bo->vmap, map_ofs +XE_PAGE_SIZE * level, u64, entry | flags); }@@ -247,7 +249,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,/* Write PDE's that point to our BO. */ for (i =0 ; i < num_entries - num_level; i++) { entry =xe_pde_encode (bo, i *XE_PAGE_SIZE , -XE_CACHE_WB ); + pat_index);xe_map_wr (xe, &bo->vmap, map_ofs +XE_PAGE_SIZE + (i +1 ) *8 , u64, entry);@@ -1256,7 +1258,7 @@ xe_migrate_update_pgtables(struct xe_migrate *m,XE_WARN_ON (pt_bo->size !=SZ_4K ); - addr =xe_pte_encode (pt_bo,0 ,XE_CACHE_WB ,0 ); + addr =xe_pte_encode (pt_bo,0 ,xe_pat_get_index (xe,XE_CACHE_WB ),0 ); bb->cs[bb->len++] =lower_32_bits (addr); bb->cs[bb->len++] =upper_32_bits (addr); }drivers/gpu/drm/xe/xe_pt.c +12
-20@@ -10,6 +10,7 @@#include "xe_gt.h" #include "xe_gt_tlb_invalidation.h" #include "xe_migrate.h" +#include "xe_pat.h" #include "xe_pt_types.h" #include "xe_pt_walk.h" #include "xe_res_cursor.h" @@ -57,24 +58,22 @@ static struct xe_pt *xe_pt_entry(struct xe_pt_dir *pt_dir, unsigned int index)* * Return: An encoded page directory entry. No errors. */ -u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level cache) +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 pat_index) {u64 pde;struct xe_vm *vm = bo->vm;struct xe_device *xe = vm->xe; - pde =xe_bo_addr (bo, bo_offset,XE_PAGE_SIZE ); pde |=XE_PAGE_PRESENT |XE_PAGE_RW ;if ((vm->pat_encode).pde_encode) - pde = (vm->pat_encode).pde_encode (xe, pde, cache); + pde = (vm->pat_encode).pde_encode (xe, pde, pat_index);return pde; } -static u64 __pte_encode (u64 pte,enum xe_cache_level cache, +static u64 __pte_encode (u64 pte,u32 pat_index,struct xe_vma *vma,u32 pt_level) {struct xe_vm *vm =xe_vma_vm (vma);@@ -89,7 +88,7 @@ static u64 __pte_encode(u64 pte, enum xe_cache_level cache, pte |=XE_PTE_NULL ;if ((vm->pat_encode).pte_encode) - pte = (vm->pat_encode).pte_encode (xe, pte, cache); + pte = (vm->pat_encode).pte_encode (xe, pte, pat_index);if (pt_level ==1 ) pte |=XE_PDE_PS_2M ;@@ -112,7 +111,7 @@ static u64 __pte_encode(u64 pte, enum xe_cache_level cache,* * Return: An encoded page-table entry. No errors. */ -u64 xe_pte_encode (struct xe_bo *bo,u64 offset,enum xe_cache_level cache, +u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 pat_index,u32 pt_level) {u64 pte;@@ -121,7 +120,7 @@ u64 xe_pte_encode(struct xe_bo *bo, u64 offset, enum xe_cache_level cache,if (xe_bo_is_vram (bo) ||xe_bo_is_stolen_devmem (bo)) pte |=XE_PPGTT_PTE_DM ; -return __pte_encode (pte, cache,NULL , pt_level); +return __pte_encode (pte, pat_index,NULL , pt_level); } static u64 __xe_pt_empty_pte(struct xe_tile *tile, struct xe_vm *vm,@@ -134,12 +133,12 @@ static u64 __xe_pt_empty_pte(struct xe_tile *tile, struct xe_vm *vm,if (level ==0 ) {u64 empty =xe_pte_encode (vm->scratch_bo[id],0 , -XE_CACHE_WB ,0 ); +xe_pat_get_index (vm->xe,XE_CACHE_WB ),0 );return empty; }else {return xe_pde_encode (vm->scratch_pt[id][level -1 ]->bo,0 , -XE_CACHE_WB ); +xe_pat_get_index (vm->xe,XE_CACHE_WB )); } }@@ -368,8 +367,6 @@ struct xe_pt_stage_bind_walk {struct xe_vm *vm;/** @tile: The tile we're building for. */ struct xe_tile *tile; -/** @cache: Desired cache level for the ptes */ -enum xe_cache_level cache;/** @default_pte: PTE flag only template. No address is associated */ u64 default_pte;/** @dma_offset: DMA offset to add to the PTE. */ @@ -604,7 +601,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset, pte =__pte_encode (is_null ?0 :xe_res_dma (curs) + xe_walk->dma_offset, - xe_walk->cache, xe_walk->vma, level); + xe_walk->vma->pat_index, xe_walk->vma, level); pte |= xe_walk->default_pte;/* @@ -669,7 +666,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset, xe_child->is_compact = true; } - pte =xe_pde_encode (xe_child->bo,0 , xe_walk->cache) | flags; + pte =xe_pde_encode (xe_child->bo,0 , xe_walk->vma->pat_index) | flags; ret =xe_pt_insert_entry (xe_walk, xe_parent, offset, xe_child, pte); }@@ -730,13 +727,8 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma,if (vma && vma->gpuva.flags &XE_VMA_ATOMIC_PTE_BIT ) xe_walk.default_pte |=XE_USM_PPGTT_PTE_AE ; xe_walk.dma_offset =vram_region_gpu_offset (bo->ttm.resource); - xe_walk.cache =XE_CACHE_WB ; - }else { -if (!xe_vma_has_no_bo (vma) && bo->flags &XE_BO_SCANOUT_BIT ) - xe_walk.cache =XE_CACHE_WT ; -else - xe_walk.cache =XE_CACHE_WB ; } +if (!xe_vma_has_no_bo (vma) &&xe_bo_is_stolen (bo)) xe_walk.dma_offset =xe_ttm_stolen_gpu_offset (xe_bo_device (bo));drivers/gpu/drm/xe/xe_pt.h +2
-4@@ -45,10 +45,8 @@ __xe_pt_unbind_vma(struct xe_tile *tile, struct xe_vma *vma, struct xe_exec_queubool xe_pt_zap_ptes (struct xe_tile *tile,struct xe_vma *vma); -u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level level); +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 pat_index); -u64 xe_pte_encode (struct xe_bo *bo,u64 offset,enum xe_cache_level cache, -u32 pt_level); +u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 pat_index,u32 pt_level);#endif drivers/gpu/drm/xe/xe_vm.c +24
-14@@ -6,6 +6,7 @@#include "xe_vm.h" #include <linux/dma-fence-array.h> +#include <linux/nospec.h> #include <drm/drm_print.h> #include <drm/ttm/ttm_execbuf_util.h> @@ -874,7 +875,8 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, u64 start, u64 end, bool read_only, bool is_null, - u8 tile_mask) + u8 tile_mask, + u32 pat_index) { struct xe_vma *vma;struct xe_tile *tile;@@ -913,6 +915,8 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, vma->tile_mask |=0x1 << id; } + vma->pat_index = pat_index; +if (vm->xe->info.platform ==XE_PVC ) vma->gpuva.flags |=XE_VMA_ATOMIC_PTE_BIT ;@@ -1194,10 +1198,8 @@ static void xe_vma_op_work_func(struct work_struct *w);static void vm_destroy_work_func (struct work_struct *w);static u64 xelp_ppgtt_pde_encode_pat (struct xe_device *xe,u64 pde_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pde_pat &= ~(XELP_PDE_PAT_MASK );if (pat_index &BIT (0 ))@@ -1213,10 +1215,8 @@ static u64 xelp_ppgtt_pde_encode_pat(struct xe_device *xe, u64 pde_pat, }static u64 xelp_ppgtt_pte_encode_pat (struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELP_PTE_PAT_MASK );if (pat_index &BIT (0 ))@@ -1622,7 +1622,7 @@ struct xe_vm *xe_vm_lookup(struct xe_file *xef, u32 id)u64 xe_vm_pdp4_descriptor (struct xe_vm *vm,struct xe_tile *tile) {return xe_pde_encode (vm->pt_root[tile->id]->bo,0 , -XE_CACHE_WB ); +xe_pat_get_index (vm->xe,XE_CACHE_WB )); } static struct dma_fence *@@ -2311,7 +2311,7 @@ static void print_op(struct xe_device *xe, struct drm_gpuva_op *op)static struct drm_gpuva_ops *vm_bind_ioctl_ops_create (struct xe_vm *vm,struct xe_bo *bo,u64 bo_offset_or_userptr,u64 addr,u64 range, -u32 operation,u8 tile_mask,u32 region) +u32 operation,u8 tile_mask,u32 region,u32 pat_index) {struct drm_gem_object *obj = bo ? &bo->ttm.base :NULL ;struct ww_acquire_ctx ww;@@ -2339,6 +2339,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo,struct xe_vma_op *op =gpuva_op_to_vma_op (__op); op->tile_mask = tile_mask; + op->pat_index = pat_index; op->map.immediate = operation &XE_VM_BIND_FLAG_IMMEDIATE ; op->map.read_only =@@ -2366,6 +2367,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo,struct xe_vma_op *op =gpuva_op_to_vma_op (__op); op->tile_mask = tile_mask; + op->pat_index = pat_index; op->prefetch.region = region; }break ;@@ -2408,7 +2410,8 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo, }static struct xe_vma *new_vma (struct xe_vm *vm,struct drm_gpuva_op_map *op, -u8 tile_mask,bool read_only,bool is_null) +u8 tile_mask,bool read_only,bool is_null, +u32 pat_index) {struct xe_bo *bo = op->gem.obj ?gem_to_xe_bo (op->gem.obj) :NULL ;struct xe_vma *vma;@@ -2425,7 +2428,7 @@ static struct xe_vma *new_vma(struct xe_vm *vm, struct drm_gpuva_op_map *op, vma =xe_vma_create (vm, bo, op->gem.offset, op->va.addr, op->va.addr + op->va.range -1 , read_only, is_null, - tile_mask); + tile_mask, pat_index);if (bo)xe_bo_unlock (bo, &ww);@@ -2539,7 +2542,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, &op->base.map, op->tile_mask, op->map.read_only, - op->map.is_null); + op->map.is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2567,7 +2570,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, op->base.remap.prev, op->tile_mask, read_only, - is_null); + is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2603,7 +2606,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, op->base.remap.next, op->tile_mask, read_only, - is_null); + is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -3158,8 +3161,14 @@ static int vm_bind_ioctl_check_args(struct xe_device *xe,u32 obj = (*bind_ops)[i].obj;u64 obj_offset = (*bind_ops)[i].obj_offset;u32 region = (*bind_ops)[i].region; +u32 pat_index = (*bind_ops)[i].pat_index;bool is_null = op &XE_VM_BIND_FLAG_NULL ; +if (XE_IOCTL_DBG (xe, pat_index >= xe->info.pat_table_n_entries)) { + err = -EINVAL ; + goto free_bind_ops; + } +if (i ==0 ) { *async = !!(op &XE_VM_BIND_FLAG_ASYNC ); } else if (XE_IOCTL_DBG(xe, !*async) ||@@ -3346,8 +3355,25 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)struct drm_gem_object *gem_obj;u64 range = bind_ops[i].range;u64 addr = bind_ops[i].addr; +u32 op = bind_ops[i].op;u32 obj = bind_ops[i].obj;u64 obj_offset = bind_ops[i].obj_offset;
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) · line 3370
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c +10
-0+u32 pat_index = bind_ops[i].pat_index; +u16 coh_mode; + + pat_index =array_index_nospec (pat_index, + xe->info.pat_table_n_entries); + coh_mode =xe_pat_index_get_coh_mode (xe, pat_index); +if (XE_IOCTL_DBG (xe, !coh_mode)) { + err = -EINVAL ; + goto put_obj; + }
In my opinion this check belongs to vm_bind_ioctl_check_args().
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) · line 3376
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +6
-0+ +if (XE_IOCTL_DBG (xe,VM_BIND_OP (op) ==XE_VM_BIND_OP_MAP_USERPTR && + coh_mode ==XE_GEM_COHERENCY_NONE )) { + err = -EINVAL ; + goto put_obj; + }
Same for this one.
Quoted 44 lines
drivers/gpu/drm/xe/xe_vm.c +18
-1if (!obj)continue ;@@ -3375,6 +3401,22 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) goto put_obj; } } + +if (bos[i]->coh_mode) { +if (XE_IOCTL_DBG (xe, bos[i]->coh_mode != coh_mode)) { + err = -EINVAL ; + goto put_obj; + } + }else if (XE_IOCTL_DBG (xe, coh_mode ==XE_GEM_COHERENCY_NONE )) { +/* +* Imported dma-buf from a different device should +* require 1way or 2way coherency since we don't know +* how it was mapped on CPU. Just assume is it +* potentially cached on CPU side. +*/ + err = -EINVAL ; + goto put_obj; + } }if (args->num_syncs) {@@ -3412,10 +3454,11 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)u64 obj_offset = bind_ops[i].obj_offset;u8 tile_mask = bind_ops[i].tile_mask;u32 region = bind_ops[i].region; +u32 pat_index = bind_ops[i].pat_index; ops[i] =vm_bind_ioctl_ops_create (vm, bos[i], obj_offset, addr, range, op, tile_mask, - region); + region, pat_index);if (IS_ERR (ops[i])) { err =PTR_ERR (ops[i]); ops[i] =NULL ;drivers/gpu/drm/xe/xe_vm_types.h +0
-0
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vma { · line 117
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm_types.h +4
-0@@ -111,6 +111,11 @@ struct xe_vma {*/ u8 tile_present; +/** +* @pat_index: The pat index to use when encoding the PTEs for this vma. +*/ +u32 pat_index;
u32 is overkill for pat_index.
Quoted 78 lines
drivers/gpu/drm/xe/xe_vm_types.h +5
-4+struct {struct list_head rebind_link; } notifier;@@ -338,10 +343,8 @@ struct xe_vm {bool batch_invalidate_tlb;struct { -u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache); -u64 (*pde_encode)(struct xe_device *xe,u64 pde_pat, -enum xe_cache_level cache); +u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat,u32 pat_index); +u64 (*pde_encode)(struct xe_device *xe,u64 pde_pat,u32 pat_index); } pat_encode; };@@ -417,6 +420,8 @@ struct xe_vma_op {struct async_op_fence *fence;/** @tile_mask: gt mask for this operation */ u8 tile_mask; +/** @pat_index: The pat index to use for this operation. */ +u32 pat_index;/** @flags: operation flags */ enum xe_vma_op_flags flags;include/uapi/drm/xe_drm.h +39
-2@@ -600,8 +600,45 @@ struct drm_xe_vm_bind_op {*/ __u32 obj; -/** @pad: MBZ */ -__u32 pad; +/** +* @pat_index: The platform defined @pat_index to use for this mapping. +* The index basically maps to some predefined memory attributes, +* including things like caching, coherency and likely other stuff in +* the future. The exact meaning of the pat_index is platform specific +* and defined in the Bspec and PRMs. When the KMD sets up the binding +* the index here is encoded into the ppGTT PTE. +* +* For coherency the @pat_index needs to match the +* drm_xe_gem_create.coh_mode, so either XE_GEM_COHERENCY_NONE, +* XE_GEM_COHERENCY_1WAY or XE_GEM_COHERENCY_2WAY. The KMD will extract +* the coherency mode from the @pat_index and reject if there is a +* mismatch (see note below for pre-MTL platforms). +* +* Note: On pre-MTL platforms there is only a caching mode and no +* explicit coherency mode, but on such hardware there is always a +* shared-LLC (or is dgpu) so all GT memory accesses are coherent with +* CPU caches even with the caching mode set as uncached. It's only the +* display engine that is incoherent (on dgpu it must be in VRAM which +* is always mapped as WC on the CPU). However to keep the uapi somewhat +* consistent with newer platforms the KMD groups the different cache +* levels into the following coherency buckets on all pre-MTL platforms: +* +* ppGTT UC -> XE_GEM_COHERENCY_NONE +* ppGTT WC -> XE_GEM_COHERENCY_NONE +* ppGTT WT -> XE_GEM_COHERENCY_NONE +* ppGTT WB -> XE_GEM_COHERENCY_2WAY +* +* In practice UC/WC/WT should only ever used for scanout surfaces on +* such platforms since it is only the display engine that is actually +* incoherent. Everything else should typically use WB given that we +* have a shared-LLC. On MTL+ this completely changes (also potentially +* no shared-LLC) and the HW defines the coherency mode as part of the +* @pat_index. +* +* Note: For userptr and externally imported dma-buf the kernel expects +* either 1WAY or 2WAY for the @pat_index. +*/ +__u32 pat_index;union {/**
Re: [Intel-xe] [RFC 5/5] drm/xe/uapi: support pat_index selection with vm_bind
Quoted 501 lines · Jose
Quoted 487 lines · Matthew Auld
Allow userspace to directly control the
pat_indexfor a given vm binding. This should allow directly controlling the coherency, caching and potentially other stuff in the future for the ppGTT binding.The exact meaning behind the
pat_indexis very platform specific (see BSpec or PRMs) but effectively maps to some predefined memory attributes. From the KMD pov we only care about the coherency that is provided by thepat_index, which falls into either NONE, 1WAY or 2WAY. Thevm_bindcoherency mode for the givenpat_indexneeds to match the givencoh_modethat was set at object creation. For platforms that lack the explicit coherency mode, we treat UC/WT/WC as NONE and WB as 2WAY.For userptr mappings we lack a corresponding gem object, so the expected coherency mode is instead implicit and must fall into either 1WAY or 2WAY. Trying to use NONE will be rejected by the kernel. For imported dma-buf (from a different device) the coherency mode is also implicit and must also be either 1WAY or 2WAY.
As part of adding
pat_indexsupport withvm_bindwe also need stop usingxe_cache_leveland instead use thepat_indexin various places. We still make use ofxe_cache_level, but only as a convenience for kernel internal objectsi (internally it maps to some reasonablepat_index). For now this is just a 1:1 conversion of the existing code, however for platforms like MTL+ we might need to give more control throughbo_createor stop using WB on the CPU side if we need CPU access.Bspec: 45101, 44235 #xe
Bspec: 70552, 71582, 59400 #xe2Signed-off-by Matthew Auld <[email protected]>Cc Pallavi Mishra <[email protected]>Cc Thomas Hellström <[email protected]>Cc Joonas Lahtinen <[email protected]>Cc Lucas De Marchi <[email protected]>Cc Matt Roper <[email protected]>Cc José Roberto de Souza <[email protected]>Cc Filip Hazubski <[email protected]>Cc Carl Zhang <[email protected]>Cc Effie Yu <[email protected]>--- drivers/gpu/drm/i915/gt/intel_gtt.h | 2 +- drivers/gpu/drm/xe/tests/xe_migrate.c | 2 +- drivers/gpu/drm/xe/xe_ggtt.c | 7 ++- drivers/gpu/drm/xe/xe_ggtt_types.h | 2 +- drivers/gpu/drm/xe/xe_migrate.c | 14 ++--- drivers/gpu/drm/xe/xe_pt.c | 32 +++++------- drivers/gpu/drm/xe/xe_pt.h | 6 +-- drivers/gpu/drm/xe/xe_vm.c | 73 +++++++++++++++++++++------ drivers/gpu/drm/xe/xe_vm_types.h | 13 +++-- include/uapi/drm/xe_drm.h | 41 ++++++++++++++- 10 files changed, 134 insertions(+), 58 deletions(-)drivers/gpu/drm/i915/gt/intel_gtt.h +1
-1@@ -302,7 +302,7 @@ struct i915_address_space { (*alloc_scratch_dma)(struct i915_address_space *vm, int sz);u64 (*pte_encode)(dma_addr_t addr, -unsigned int pat_index, +u32 pat_index,u32 flags);/* Create a valid PTE */ #define PTE_READ_ONLY BIT(0)#define PTE_LM BIT(1)drivers/gpu/drm/xe/tests/xe_migrate.c +1
-1@@ -301,7 +301,7 @@ static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test)/* First part of the test, are we updating our pagetable bo with a new entry? */ xe_map_wr (xe, &bo->vmap,XE_PAGE_SIZE * (NUM_KERNEL_PDE -1 ), u64,0xdeaddeadbeefbeef ); - expected =xe_pte_encode (pt,0 ,XE_CACHE_WB ,0 ); + expected =xe_pte_encode (pt,0 ,xe_pat_get_index (xe,XE_CACHE_WB ),0 );if (m->q->vm->flags &XE_VM_FLAG_64K ) expected |=XE_PTE_PS64 ;if (xe_bo_is_vram (pt))drivers/gpu/drm/xe/xe_ggtt.c +3
-4@@ -41,7 +41,8 @@ u64 xe_ggtt_pte_encode(struct xe_bo *bo, u64 bo_offset) pte |=XE_GGTT_PTE_DM ;if ((ggtt->pat_encode).pte_encode) - pte = (ggtt->pat_encode).pte_encode (xe, pte,XE_CACHE_WB_1_WAY ); + pte = (ggtt->pat_encode).pte_encode (xe, pte, +xe_pat_get_index (xe,XE_CACHE_WB_1_WAY ));return pte; }@@ -102,10 +103,8 @@ static void primelockdep(struct xe_ggtt *ggtt) }static u64 xelpg_ggtt_pte_encode_pat (struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELPG_GGTT_PTE_PAT_MASK );if (pat_index &BIT (0 ))drivers/gpu/drm/xe/xe_ggtt_types.h +1
-1@@ -31,7 +31,7 @@ struct xe_ggtt {struct {u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache); +u32 pat_index); } pat_encode; };drivers/gpu/drm/xe/xe_migrate.c +8
-6@@ -24,6 +24,7 @@#include "xe_lrc.h" #include "xe_map.h" #include "xe_mocs.h" +#include "xe_pat.h" #include "xe_pt.h" #include "xe_res_cursor.h" #include "xe_sched_job.h" @@ -162,6 +163,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,u32 num_entries =NUM_PT_SLOTS , num_level = vm->pt_root[id]->level;u32 map_ofs, level, i;struct xe_bo *bo, *batch = tile->mem.kernel_bb_pool->bo; +u32 pat_index =xe_pat_get_index (xe,XE_CACHE_WB );u64 entry;int ret;@@ -189,14 +191,14 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,return ret; } - entry =xe_pde_encode (bo, bo->size -XE_PAGE_SIZE ,XE_CACHE_WB ); + entry =xe_pde_encode (bo, bo->size -XE_PAGE_SIZE , pat_index);xe_pt_write (xe, &vm->pt_root[id]->bo->vmap,0 , entry); map_ofs = (num_entries - num_level) *XE_PAGE_SIZE ;/* Map the entire BO in our level 0 pt */ for (i =0 , level =0 ; i < num_entries; level++) { - entry =xe_pte_encode (bo, i *XE_PAGE_SIZE ,XE_CACHE_WB ,0 ); + entry =xe_pte_encode (bo, i *XE_PAGE_SIZE , pat_index,0 );xe_map_wr (xe, &bo->vmap, map_ofs + level *8 , u64, entry);@@ -214,7 +216,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,for (i =0 ; i < batch->size; i += vm->flags &XE_VM_FLAG_64K ?XE_64K_PAGE_SIZE :XE_PAGE_SIZE ) { - entry =xe_pte_encode (batch, i,XE_CACHE_WB ,0 ); + entry =xe_pte_encode (batch, i, pat_index,0 );xe_map_wr (xe, &bo->vmap, map_ofs + level *8 , u64, entry);@@ -239,7 +241,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m, flags =XE_PDE_64K ; entry =xe_pde_encode (bo, map_ofs + (level -1 ) * -XE_PAGE_SIZE ,XE_CACHE_WB ); +XE_PAGE_SIZE , pat_index);xe_map_wr (xe, &bo->vmap, map_ofs +XE_PAGE_SIZE * level, u64, entry | flags); }@@ -247,7 +249,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,/* Write PDE's that point to our BO. */ for (i =0 ; i < num_entries - num_level; i++) { entry =xe_pde_encode (bo, i *XE_PAGE_SIZE , -XE_CACHE_WB ); + pat_index);xe_map_wr (xe, &bo->vmap, map_ofs +XE_PAGE_SIZE + (i +1 ) *8 , u64, entry);@@ -1256,7 +1258,7 @@ xe_migrate_update_pgtables(struct xe_migrate *m,XE_WARN_ON (pt_bo->size !=SZ_4K ); - addr =xe_pte_encode (pt_bo,0 ,XE_CACHE_WB ,0 ); + addr =xe_pte_encode (pt_bo,0 ,xe_pat_get_index (xe,XE_CACHE_WB ),0 ); bb->cs[bb->len++] =lower_32_bits (addr); bb->cs[bb->len++] =upper_32_bits (addr); }drivers/gpu/drm/xe/xe_pt.c +12
-20@@ -10,6 +10,7 @@#include "xe_gt.h" #include "xe_gt_tlb_invalidation.h" #include "xe_migrate.h" +#include "xe_pat.h" #include "xe_pt_types.h" #include "xe_pt_walk.h" #include "xe_res_cursor.h" @@ -57,24 +58,22 @@ static struct xe_pt *xe_pt_entry(struct xe_pt_dir *pt_dir, unsigned int index)* * Return: An encoded page directory entry. No errors. */ -u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level cache) +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 pat_index) {u64 pde;struct xe_vm *vm = bo->vm;struct xe_device *xe = vm->xe; - pde =xe_bo_addr (bo, bo_offset,XE_PAGE_SIZE ); pde |=XE_PAGE_PRESENT |XE_PAGE_RW ;if ((vm->pat_encode).pde_encode) - pde = (vm->pat_encode).pde_encode (xe, pde, cache); + pde = (vm->pat_encode).pde_encode (xe, pde, pat_index);return pde; } -static u64 __pte_encode (u64 pte,enum xe_cache_level cache, +static u64 __pte_encode (u64 pte,u32 pat_index,struct xe_vma *vma,u32 pt_level) {struct xe_vm *vm =xe_vma_vm (vma);@@ -89,7 +88,7 @@ static u64 __pte_encode(u64 pte, enum xe_cache_level cache, pte |=XE_PTE_NULL ;if ((vm->pat_encode).pte_encode) - pte = (vm->pat_encode).pte_encode (xe, pte, cache); + pte = (vm->pat_encode).pte_encode (xe, pte, pat_index);if (pt_level ==1 ) pte |=XE_PDE_PS_2M ;@@ -112,7 +111,7 @@ static u64 __pte_encode(u64 pte, enum xe_cache_level cache,* * Return: An encoded page-table entry. No errors. */ -u64 xe_pte_encode (struct xe_bo *bo,u64 offset,enum xe_cache_level cache, +u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 pat_index,u32 pt_level) {u64 pte;@@ -121,7 +120,7 @@ u64 xe_pte_encode(struct xe_bo *bo, u64 offset, enum xe_cache_level cache,if (xe_bo_is_vram (bo) ||xe_bo_is_stolen_devmem (bo)) pte |=XE_PPGTT_PTE_DM ; -return __pte_encode (pte, cache,NULL , pt_level); +return __pte_encode (pte, pat_index,NULL , pt_level); } static u64 __xe_pt_empty_pte(struct xe_tile *tile, struct xe_vm *vm,@@ -134,12 +133,12 @@ static u64 __xe_pt_empty_pte(struct xe_tile *tile, struct xe_vm *vm,if (level ==0 ) {u64 empty =xe_pte_encode (vm->scratch_bo[id],0 , -XE_CACHE_WB ,0 ); +xe_pat_get_index (vm->xe,XE_CACHE_WB ),0 );return empty; }else {return xe_pde_encode (vm->scratch_pt[id][level -1 ]->bo,0 , -XE_CACHE_WB ); +xe_pat_get_index (vm->xe,XE_CACHE_WB )); } }@@ -368,8 +367,6 @@ struct xe_pt_stage_bind_walk {struct xe_vm *vm;/** @tile: The tile we're building for. */ struct xe_tile *tile; -/** @cache: Desired cache level for the ptes */ -enum xe_cache_level cache;/** @default_pte: PTE flag only template. No address is associated */ u64 default_pte;/** @dma_offset: DMA offset to add to the PTE. */ @@ -604,7 +601,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset, pte =__pte_encode (is_null ?0 :xe_res_dma (curs) + xe_walk->dma_offset, - xe_walk->cache, xe_walk->vma, level); + xe_walk->vma->pat_index, xe_walk->vma, level); pte |= xe_walk->default_pte;/* @@ -669,7 +666,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset, xe_child->is_compact = true; } - pte =xe_pde_encode (xe_child->bo,0 , xe_walk->cache) | flags; + pte =xe_pde_encode (xe_child->bo,0 , xe_walk->vma->pat_index) | flags; ret =xe_pt_insert_entry (xe_walk, xe_parent, offset, xe_child, pte); }@@ -730,13 +727,8 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma,if (vma && vma->gpuva.flags &XE_VMA_ATOMIC_PTE_BIT ) xe_walk.default_pte |=XE_USM_PPGTT_PTE_AE ; xe_walk.dma_offset =vram_region_gpu_offset (bo->ttm.resource); - xe_walk.cache =XE_CACHE_WB ; - }else { -if (!xe_vma_has_no_bo (vma) && bo->flags &XE_BO_SCANOUT_BIT ) - xe_walk.cache =XE_CACHE_WT ; -else - xe_walk.cache =XE_CACHE_WB ; } +if (!xe_vma_has_no_bo (vma) &&xe_bo_is_stolen (bo)) xe_walk.dma_offset =xe_ttm_stolen_gpu_offset (xe_bo_device (bo));drivers/gpu/drm/xe/xe_pt.h +2
-4@@ -45,10 +45,8 @@ __xe_pt_unbind_vma(struct xe_tile *tile, struct xe_vma *vma, struct xe_exec_queubool xe_pt_zap_ptes (struct xe_tile *tile,struct xe_vma *vma); -u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level level); +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 pat_index); -u64 xe_pte_encode (struct xe_bo *bo,u64 offset,enum xe_cache_level cache, -u32 pt_level); +u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 pat_index,u32 pt_level);#endif drivers/gpu/drm/xe/xe_vm.c +24
-14@@ -6,6 +6,7 @@#include "xe_vm.h" #include <linux/dma-fence-array.h> +#include <linux/nospec.h> #include <drm/drm_print.h> #include <drm/ttm/ttm_execbuf_util.h> @@ -874,7 +875,8 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, u64 start, u64 end, bool read_only, bool is_null, - u8 tile_mask) + u8 tile_mask, + u32 pat_index) { struct xe_vma *vma;struct xe_tile *tile;@@ -913,6 +915,8 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, vma->tile_mask |=0x1 << id; } + vma->pat_index = pat_index; +if (vm->xe->info.platform ==XE_PVC ) vma->gpuva.flags |=XE_VMA_ATOMIC_PTE_BIT ;@@ -1194,10 +1198,8 @@ static void xe_vma_op_work_func(struct work_struct *w);static void vm_destroy_work_func (struct work_struct *w);static u64 xelp_ppgtt_pde_encode_pat (struct xe_device *xe,u64 pde_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pde_pat &= ~(XELP_PDE_PAT_MASK );if (pat_index &BIT (0 ))@@ -1213,10 +1215,8 @@ static u64 xelp_ppgtt_pde_encode_pat(struct xe_device *xe, u64 pde_pat, }static u64 xelp_ppgtt_pte_encode_pat (struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELP_PTE_PAT_MASK );if (pat_index &BIT (0 ))@@ -1622,7 +1622,7 @@ struct xe_vm *xe_vm_lookup(struct xe_file *xef, u32 id)u64 xe_vm_pdp4_descriptor (struct xe_vm *vm,struct xe_tile *tile) {return xe_pde_encode (vm->pt_root[tile->id]->bo,0 , -XE_CACHE_WB ); +xe_pat_get_index (vm->xe,XE_CACHE_WB )); } static struct dma_fence *@@ -2311,7 +2311,7 @@ static void print_op(struct xe_device *xe, struct drm_gpuva_op *op)static struct drm_gpuva_ops *vm_bind_ioctl_ops_create (struct xe_vm *vm,struct xe_bo *bo,u64 bo_offset_or_userptr,u64 addr,u64 range, -u32 operation,u8 tile_mask,u32 region) +u32 operation,u8 tile_mask,u32 region,u32 pat_index) {struct drm_gem_object *obj = bo ? &bo->ttm.base :NULL ;struct ww_acquire_ctx ww;@@ -2339,6 +2339,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo,struct xe_vma_op *op =gpuva_op_to_vma_op (__op); op->tile_mask = tile_mask; + op->pat_index = pat_index; op->map.immediate = operation &XE_VM_BIND_FLAG_IMMEDIATE ; op->map.read_only =@@ -2366,6 +2367,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo,struct xe_vma_op *op =gpuva_op_to_vma_op (__op); op->tile_mask = tile_mask; + op->pat_index = pat_index; op->prefetch.region = region; }break ;@@ -2408,7 +2410,8 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo, }static struct xe_vma *new_vma (struct xe_vm *vm,struct drm_gpuva_op_map *op, -u8 tile_mask,bool read_only,bool is_null) +u8 tile_mask,bool read_only,bool is_null, +u32 pat_index) {struct xe_bo *bo = op->gem.obj ?gem_to_xe_bo (op->gem.obj) :NULL ;struct xe_vma *vma;@@ -2425,7 +2428,7 @@ static struct xe_vma *new_vma(struct xe_vm *vm, struct drm_gpuva_op_map *op, vma =xe_vma_create (vm, bo, op->gem.offset, op->va.addr, op->va.addr + op->va.range -1 , read_only, is_null, - tile_mask); + tile_mask, pat_index);if (bo)xe_bo_unlock (bo, &ww);@@ -2539,7 +2542,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, &op->base.map, op->tile_mask, op->map.read_only, - op->map.is_null); + op->map.is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2567,7 +2570,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, op->base.remap.prev, op->tile_mask, read_only, - is_null); + is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2603,7 +2606,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, op->base.remap.next, op->tile_mask, read_only, - is_null); + is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -3158,8 +3161,14 @@ static int vm_bind_ioctl_check_args(struct xe_device *xe,u32 obj = (*bind_ops)[i].obj;u64 obj_offset = (*bind_ops)[i].obj_offset;u32 region = (*bind_ops)[i].region; +u32 pat_index = (*bind_ops)[i].pat_index;bool is_null = op &XE_VM_BIND_FLAG_NULL ; +if (XE_IOCTL_DBG (xe, pat_index >= xe->info.pat_table_n_entries)) { + err = -EINVAL ; + goto free_bind_ops; + } +if (i ==0 ) { *async = !!(op &XE_VM_BIND_FLAG_ASYNC ); } else if (XE_IOCTL_DBG(xe, !*async) ||@@ -3346,8 +3355,25 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)struct drm_gem_object *gem_obj;u64 range = bind_ops[i].range;u64 addr = bind_ops[i].addr; +u32 op = bind_ops[i].op;u32 obj = bind_ops[i].obj;u64 obj_offset = bind_ops[i].obj_offset;drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) · line 3370
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c +10
-0+u32 pat_index = bind_ops[i].pat_index; +u16 coh_mode; + + pat_index =array_index_nospec (pat_index, + xe->info.pat_table_n_entries); + coh_mode =xe_pat_index_get_coh_mode (xe, pat_index); +if (XE_IOCTL_DBG (xe, !coh_mode)) { + err = -EINVAL ; + goto put_obj; + }In my opinion this check belongs to
vm_bind_ioctl_check_args().
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) · line 3376
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +6
-0+ +if (XE_IOCTL_DBG (xe,VM_BIND_OP (op) ==XE_VM_BIND_OP_MAP_USERPTR && + coh_mode ==XE_GEM_COHERENCY_NONE )) { + err = -EINVAL ; + goto put_obj; + }Same for this one.
OK, will take a look.
Quoted 48 lines
Quoted 48 lines
drivers/gpu/drm/xe/xe_vm.c +18
-1if (!obj)continue ;@@ -3375,6 +3401,22 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) goto put_obj; } } + +if (bos[i]->coh_mode) { +if (XE_IOCTL_DBG (xe, bos[i]->coh_mode != coh_mode)) { + err = -EINVAL ; + goto put_obj; + } + }else if (XE_IOCTL_DBG (xe, coh_mode ==XE_GEM_COHERENCY_NONE )) { +/* +* Imported dma-buf from a different device should +* require 1way or 2way coherency since we don't know +* how it was mapped on CPU. Just assume is it +* potentially cached on CPU side. +*/ + err = -EINVAL ; + goto put_obj; + } }if (args->num_syncs) {@@ -3412,10 +3454,11 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)u64 obj_offset = bind_ops[i].obj_offset;u8 tile_mask = bind_ops[i].tile_mask;u32 region = bind_ops[i].region; +u32 pat_index = bind_ops[i].pat_index; ops[i] =vm_bind_ioctl_ops_create (vm, bos[i], obj_offset, addr, range, op, tile_mask, - region); + region, pat_index);if (IS_ERR (ops[i])) { err =PTR_ERR (ops[i]); ops[i] =NULL ;drivers/gpu/drm/xe/xe_vm_types.h +0
-0@@ -111,6 +111,11 @@ struct xe_vma {*/ u8 tile_present;
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vma { · line 117
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm_types.h +4
-0+/** +* @pat_index: The pat index to use when encoding the PTEs for this vma. +*/ +u32 pat_index;u32 is overkill for
pat_index.
What about u16 instead?
Quoted 80 lines
Quoted 79 lines
drivers/gpu/drm/xe/xe_vm_types.h +5
-4+struct {struct list_head rebind_link; } notifier;@@ -338,10 +343,8 @@ struct xe_vm {bool batch_invalidate_tlb;struct { -u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache); -u64 (*pde_encode)(struct xe_device *xe,u64 pde_pat, -enum xe_cache_level cache); +u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat,u32 pat_index); +u64 (*pde_encode)(struct xe_device *xe,u64 pde_pat,u32 pat_index); } pat_encode; };@@ -417,6 +420,8 @@ struct xe_vma_op {struct async_op_fence *fence;/** @tile_mask: gt mask for this operation */ u8 tile_mask; +/** @pat_index: The pat index to use for this operation. */ +u32 pat_index;/** @flags: operation flags */ enum xe_vma_op_flags flags;include/uapi/drm/xe_drm.h +39
-2@@ -600,8 +600,45 @@ struct drm_xe_vm_bind_op {*/ __u32 obj; -/** @pad: MBZ */ -__u32 pad; +/** +* @pat_index: The platform defined @pat_index to use for this mapping. +* The index basically maps to some predefined memory attributes, +* including things like caching, coherency and likely other stuff in +* the future. The exact meaning of the pat_index is platform specific +* and defined in the Bspec and PRMs. When the KMD sets up the binding +* the index here is encoded into the ppGTT PTE. +* +* For coherency the @pat_index needs to match the +* drm_xe_gem_create.coh_mode, so either XE_GEM_COHERENCY_NONE, +* XE_GEM_COHERENCY_1WAY or XE_GEM_COHERENCY_2WAY. The KMD will extract +* the coherency mode from the @pat_index and reject if there is a +* mismatch (see note below for pre-MTL platforms). +* +* Note: On pre-MTL platforms there is only a caching mode and no +* explicit coherency mode, but on such hardware there is always a +* shared-LLC (or is dgpu) so all GT memory accesses are coherent with +* CPU caches even with the caching mode set as uncached. It's only the +* display engine that is incoherent (on dgpu it must be in VRAM which +* is always mapped as WC on the CPU). However to keep the uapi somewhat +* consistent with newer platforms the KMD groups the different cache +* levels into the following coherency buckets on all pre-MTL platforms: +* +* ppGTT UC -> XE_GEM_COHERENCY_NONE +* ppGTT WC -> XE_GEM_COHERENCY_NONE +* ppGTT WT -> XE_GEM_COHERENCY_NONE +* ppGTT WB -> XE_GEM_COHERENCY_2WAY +* +* In practice UC/WC/WT should only ever used for scanout surfaces on +* such platforms since it is only the display engine that is actually +* incoherent. Everything else should typically use WB given that we +* have a shared-LLC. On MTL+ this completely changes (also potentially +* no shared-LLC) and the HW defines the coherency mode as part of the +* @pat_index. +* +* Note: For userptr and externally imported dma-buf the kernel expects +* either 1WAY or 2WAY for the @pat_index. +*/ +__u32 pat_index;union {/**
Re: [Intel-xe] [RFC 5/5] drm/xe/uapi: support pat_index selection with vm_bind
In reply to Matthew AuldQuoted 18 lines · Matthew Auld
Allow userspace to directly control the
pat_indexfor a given vm binding. This should allow directly controlling the coherency, caching and potentially other stuff in the future for the ppGTT binding.The exact meaning behind the
pat_indexis very platform specific (see BSpec or PRMs) but effectively maps to some predefined memory attributes. From the KMD pov we only care about the coherency that is provided by thepat_index, which falls into either NONE, 1WAY or 2WAY. Thevm_bindcoherency mode for the givenpat_indexneeds to match the givencoh_modethat was set at object creation. For platforms that lack the explicit coherency mode, we treat UC/WT/WC as NONE and WB as 2WAY.For userptr mappings we lack a corresponding gem object, so the expected coherency mode is instead implicit and must fall into either 1WAY or 2WAY. Trying to use NONE will be rejected by the kernel. For imported dma-buf (from a different device) the coherency mode is also implicit and must also be either 1WAY or 2WAY.
Quoted 7 lines
As part of adding
pat_indexsupport withvm_bindwe also need stop usingxe_cache_leveland instead use thepat_indexin various places. We still make use ofxe_cache_level, but only as a convenience for kernel internal objectsi (internally it maps to some reasonablepat_index). For now this is just a 1:1 conversion of the existing code, however for platforms like MTL+ we might need to give more control throughbo_createor stop using WB on the CPU side if we need CPU access.
xe_ttm_tt_create() caching should not also take into consideration the PAT index?
Quoted 607 lines
Bspec: 45101, 44235 #xe
Bspec: 70552, 71582, 59400 #xe2Signed-off-by Matthew Auld <[email protected]>Cc Pallavi Mishra <[email protected]>Cc Thomas Hellström <[email protected]>Cc Joonas Lahtinen <[email protected]>Cc Lucas De Marchi <[email protected]>Cc Matt Roper <[email protected]>Cc José Roberto de Souza <[email protected]>Cc Filip Hazubski <[email protected]>Cc Carl Zhang <[email protected]>Cc Effie Yu <[email protected]>--- drivers/gpu/drm/i915/gt/intel_gtt.h | 2 +- drivers/gpu/drm/xe/tests/xe_migrate.c | 2 +- drivers/gpu/drm/xe/xe_ggtt.c | 7 ++- drivers/gpu/drm/xe/xe_ggtt_types.h | 2 +- drivers/gpu/drm/xe/xe_migrate.c | 14 ++--- drivers/gpu/drm/xe/xe_pt.c | 32 +++++------- drivers/gpu/drm/xe/xe_pt.h | 6 +-- drivers/gpu/drm/xe/xe_vm.c | 73 +++++++++++++++++++++------ drivers/gpu/drm/xe/xe_vm_types.h | 13 +++-- include/uapi/drm/xe_drm.h | 41 ++++++++++++++- 10 files changed, 134 insertions(+), 58 deletions(-)drivers/gpu/drm/i915/gt/intel_gtt.h +1
-1@@ -302,7 +302,7 @@ struct i915_address_space { (*alloc_scratch_dma)(struct i915_address_space *vm, int sz);u64 (*pte_encode)(dma_addr_t addr, -unsigned int pat_index, +u32 pat_index,u32 flags);/* Create a valid PTE */ #define PTE_READ_ONLY BIT(0)#define PTE_LM BIT(1)drivers/gpu/drm/xe/tests/xe_migrate.c +1
-1@@ -301,7 +301,7 @@ static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test)/* First part of the test, are we updating our pagetable bo with a new entry? */ xe_map_wr (xe, &bo->vmap,XE_PAGE_SIZE * (NUM_KERNEL_PDE -1 ), u64,0xdeaddeadbeefbeef ); - expected =xe_pte_encode (pt,0 ,XE_CACHE_WB ,0 ); + expected =xe_pte_encode (pt,0 ,xe_pat_get_index (xe,XE_CACHE_WB ),0 );if (m->q->vm->flags &XE_VM_FLAG_64K ) expected |=XE_PTE_PS64 ;if (xe_bo_is_vram (pt))drivers/gpu/drm/xe/xe_ggtt.c +3
-4@@ -41,7 +41,8 @@ u64 xe_ggtt_pte_encode(struct xe_bo *bo, u64 bo_offset) pte |=XE_GGTT_PTE_DM ;if ((ggtt->pat_encode).pte_encode) - pte = (ggtt->pat_encode).pte_encode (xe, pte,XE_CACHE_WB_1_WAY ); + pte = (ggtt->pat_encode).pte_encode (xe, pte, +xe_pat_get_index (xe,XE_CACHE_WB_1_WAY ));return pte; }@@ -102,10 +103,8 @@ static void primelockdep(struct xe_ggtt *ggtt) }static u64 xelpg_ggtt_pte_encode_pat (struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELPG_GGTT_PTE_PAT_MASK );if (pat_index &BIT (0 ))drivers/gpu/drm/xe/xe_ggtt_types.h +1
-1@@ -31,7 +31,7 @@ struct xe_ggtt {struct {u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache); +u32 pat_index); } pat_encode; };drivers/gpu/drm/xe/xe_migrate.c +8
-6@@ -24,6 +24,7 @@#include "xe_lrc.h" #include "xe_map.h" #include "xe_mocs.h" +#include "xe_pat.h" #include "xe_pt.h" #include "xe_res_cursor.h" #include "xe_sched_job.h" @@ -162,6 +163,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,u32 num_entries =NUM_PT_SLOTS , num_level = vm->pt_root[id]->level;u32 map_ofs, level, i;struct xe_bo *bo, *batch = tile->mem.kernel_bb_pool->bo; +u32 pat_index =xe_pat_get_index (xe,XE_CACHE_WB );u64 entry;int ret;@@ -189,14 +191,14 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,return ret; } - entry =xe_pde_encode (bo, bo->size -XE_PAGE_SIZE ,XE_CACHE_WB ); + entry =xe_pde_encode (bo, bo->size -XE_PAGE_SIZE , pat_index);xe_pt_write (xe, &vm->pt_root[id]->bo->vmap,0 , entry); map_ofs = (num_entries - num_level) *XE_PAGE_SIZE ;/* Map the entire BO in our level 0 pt */ for (i =0 , level =0 ; i < num_entries; level++) { - entry =xe_pte_encode (bo, i *XE_PAGE_SIZE ,XE_CACHE_WB ,0 ); + entry =xe_pte_encode (bo, i *XE_PAGE_SIZE , pat_index,0 );xe_map_wr (xe, &bo->vmap, map_ofs + level *8 , u64, entry);@@ -214,7 +216,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,for (i =0 ; i < batch->size; i += vm->flags &XE_VM_FLAG_64K ?XE_64K_PAGE_SIZE :XE_PAGE_SIZE ) { - entry =xe_pte_encode (batch, i,XE_CACHE_WB ,0 ); + entry =xe_pte_encode (batch, i, pat_index,0 );xe_map_wr (xe, &bo->vmap, map_ofs + level *8 , u64, entry);@@ -239,7 +241,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m, flags =XE_PDE_64K ; entry =xe_pde_encode (bo, map_ofs + (level -1 ) * -XE_PAGE_SIZE ,XE_CACHE_WB ); +XE_PAGE_SIZE , pat_index);xe_map_wr (xe, &bo->vmap, map_ofs +XE_PAGE_SIZE * level, u64, entry | flags); }@@ -247,7 +249,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,/* Write PDE's that point to our BO. */ for (i =0 ; i < num_entries - num_level; i++) { entry =xe_pde_encode (bo, i *XE_PAGE_SIZE , -XE_CACHE_WB ); + pat_index);xe_map_wr (xe, &bo->vmap, map_ofs +XE_PAGE_SIZE + (i +1 ) *8 , u64, entry);@@ -1256,7 +1258,7 @@ xe_migrate_update_pgtables(struct xe_migrate *m,XE_WARN_ON (pt_bo->size !=SZ_4K ); - addr =xe_pte_encode (pt_bo,0 ,XE_CACHE_WB ,0 ); + addr =xe_pte_encode (pt_bo,0 ,xe_pat_get_index (xe,XE_CACHE_WB ),0 ); bb->cs[bb->len++] =lower_32_bits (addr); bb->cs[bb->len++] =upper_32_bits (addr); }drivers/gpu/drm/xe/xe_pt.c +12
-20@@ -10,6 +10,7 @@#include "xe_gt.h" #include "xe_gt_tlb_invalidation.h" #include "xe_migrate.h" +#include "xe_pat.h" #include "xe_pt_types.h" #include "xe_pt_walk.h" #include "xe_res_cursor.h" @@ -57,24 +58,22 @@ static struct xe_pt *xe_pt_entry(struct xe_pt_dir *pt_dir, unsigned int index)* * Return: An encoded page directory entry. No errors. */ -u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level cache) +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 pat_index) {u64 pde;struct xe_vm *vm = bo->vm;struct xe_device *xe = vm->xe; - pde =xe_bo_addr (bo, bo_offset,XE_PAGE_SIZE ); pde |=XE_PAGE_PRESENT |XE_PAGE_RW ;if ((vm->pat_encode).pde_encode) - pde = (vm->pat_encode).pde_encode (xe, pde, cache); + pde = (vm->pat_encode).pde_encode (xe, pde, pat_index);return pde; } -static u64 __pte_encode (u64 pte,enum xe_cache_level cache, +static u64 __pte_encode (u64 pte,u32 pat_index,struct xe_vma *vma,u32 pt_level) {struct xe_vm *vm =xe_vma_vm (vma);@@ -89,7 +88,7 @@ static u64 __pte_encode(u64 pte, enum xe_cache_level cache, pte |=XE_PTE_NULL ;if ((vm->pat_encode).pte_encode) - pte = (vm->pat_encode).pte_encode (xe, pte, cache); + pte = (vm->pat_encode).pte_encode (xe, pte, pat_index);if (pt_level ==1 ) pte |=XE_PDE_PS_2M ;@@ -112,7 +111,7 @@ static u64 __pte_encode(u64 pte, enum xe_cache_level cache,* * Return: An encoded page-table entry. No errors. */ -u64 xe_pte_encode (struct xe_bo *bo,u64 offset,enum xe_cache_level cache, +u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 pat_index,u32 pt_level) {u64 pte;@@ -121,7 +120,7 @@ u64 xe_pte_encode(struct xe_bo *bo, u64 offset, enum xe_cache_level cache,if (xe_bo_is_vram (bo) ||xe_bo_is_stolen_devmem (bo)) pte |=XE_PPGTT_PTE_DM ; -return __pte_encode (pte, cache,NULL , pt_level); +return __pte_encode (pte, pat_index,NULL , pt_level); } static u64 __xe_pt_empty_pte(struct xe_tile *tile, struct xe_vm *vm,@@ -134,12 +133,12 @@ static u64 __xe_pt_empty_pte(struct xe_tile *tile, struct xe_vm *vm,if (level ==0 ) {u64 empty =xe_pte_encode (vm->scratch_bo[id],0 , -XE_CACHE_WB ,0 ); +xe_pat_get_index (vm->xe,XE_CACHE_WB ),0 );return empty; }else {return xe_pde_encode (vm->scratch_pt[id][level -1 ]->bo,0 , -XE_CACHE_WB ); +xe_pat_get_index (vm->xe,XE_CACHE_WB )); } }@@ -368,8 +367,6 @@ struct xe_pt_stage_bind_walk {struct xe_vm *vm;/** @tile: The tile we're building for. */ struct xe_tile *tile; -/** @cache: Desired cache level for the ptes */ -enum xe_cache_level cache;/** @default_pte: PTE flag only template. No address is associated */ u64 default_pte;/** @dma_offset: DMA offset to add to the PTE. */ @@ -604,7 +601,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset, pte =__pte_encode (is_null ?0 :xe_res_dma (curs) + xe_walk->dma_offset, - xe_walk->cache, xe_walk->vma, level); + xe_walk->vma->pat_index, xe_walk->vma, level); pte |= xe_walk->default_pte;/* @@ -669,7 +666,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset, xe_child->is_compact = true; } - pte =xe_pde_encode (xe_child->bo,0 , xe_walk->cache) | flags; + pte =xe_pde_encode (xe_child->bo,0 , xe_walk->vma->pat_index) | flags; ret =xe_pt_insert_entry (xe_walk, xe_parent, offset, xe_child, pte); }@@ -730,13 +727,8 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma,if (vma && vma->gpuva.flags &XE_VMA_ATOMIC_PTE_BIT ) xe_walk.default_pte |=XE_USM_PPGTT_PTE_AE ; xe_walk.dma_offset =vram_region_gpu_offset (bo->ttm.resource); - xe_walk.cache =XE_CACHE_WB ; - }else { -if (!xe_vma_has_no_bo (vma) && bo->flags &XE_BO_SCANOUT_BIT ) - xe_walk.cache =XE_CACHE_WT ; -else - xe_walk.cache =XE_CACHE_WB ; } +if (!xe_vma_has_no_bo (vma) &&xe_bo_is_stolen (bo)) xe_walk.dma_offset =xe_ttm_stolen_gpu_offset (xe_bo_device (bo));drivers/gpu/drm/xe/xe_pt.h +2
-4@@ -45,10 +45,8 @@ __xe_pt_unbind_vma(struct xe_tile *tile, struct xe_vma *vma, struct xe_exec_queubool xe_pt_zap_ptes (struct xe_tile *tile,struct xe_vma *vma); -u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level level); +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 pat_index); -u64 xe_pte_encode (struct xe_bo *bo,u64 offset,enum xe_cache_level cache, -u32 pt_level); +u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 pat_index,u32 pt_level);#endif drivers/gpu/drm/xe/xe_vm.c +58
-15@@ -6,6 +6,7 @@#include "xe_vm.h" #include <linux/dma-fence-array.h> +#include <linux/nospec.h> #include <drm/drm_print.h> #include <drm/ttm/ttm_execbuf_util.h> @@ -874,7 +875,8 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, u64 start, u64 end, bool read_only, bool is_null, - u8 tile_mask) + u8 tile_mask, + u32 pat_index) { struct xe_vma *vma;struct xe_tile *tile;@@ -913,6 +915,8 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, vma->tile_mask |=0x1 << id; } + vma->pat_index = pat_index; +if (vm->xe->info.platform ==XE_PVC ) vma->gpuva.flags |=XE_VMA_ATOMIC_PTE_BIT ;@@ -1194,10 +1198,8 @@ static void xe_vma_op_work_func(struct work_struct *w);static void vm_destroy_work_func (struct work_struct *w);static u64 xelp_ppgtt_pde_encode_pat (struct xe_device *xe,u64 pde_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pde_pat &= ~(XELP_PDE_PAT_MASK );if (pat_index &BIT (0 ))@@ -1213,10 +1215,8 @@ static u64 xelp_ppgtt_pde_encode_pat(struct xe_device *xe, u64 pde_pat, }static u64 xelp_ppgtt_pte_encode_pat (struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELP_PTE_PAT_MASK );if (pat_index &BIT (0 ))@@ -1622,7 +1622,7 @@ struct xe_vm *xe_vm_lookup(struct xe_file *xef, u32 id)u64 xe_vm_pdp4_descriptor (struct xe_vm *vm,struct xe_tile *tile) {return xe_pde_encode (vm->pt_root[tile->id]->bo,0 , -XE_CACHE_WB ); +xe_pat_get_index (vm->xe,XE_CACHE_WB )); } static struct dma_fence *@@ -2311,7 +2311,7 @@ static void print_op(struct xe_device *xe, struct drm_gpuva_op *op)static struct drm_gpuva_ops *vm_bind_ioctl_ops_create (struct xe_vm *vm,struct xe_bo *bo,u64 bo_offset_or_userptr,u64 addr,u64 range, -u32 operation,u8 tile_mask,u32 region) +u32 operation,u8 tile_mask,u32 region,u32 pat_index) {struct drm_gem_object *obj = bo ? &bo->ttm.base :NULL ;struct ww_acquire_ctx ww;@@ -2339,6 +2339,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo,struct xe_vma_op *op =gpuva_op_to_vma_op (__op); op->tile_mask = tile_mask; + op->pat_index = pat_index; op->map.immediate = operation &XE_VM_BIND_FLAG_IMMEDIATE ; op->map.read_only =@@ -2366,6 +2367,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo,struct xe_vma_op *op =gpuva_op_to_vma_op (__op); op->tile_mask = tile_mask; + op->pat_index = pat_index; op->prefetch.region = region; }break ;@@ -2408,7 +2410,8 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo, }static struct xe_vma *new_vma (struct xe_vm *vm,struct drm_gpuva_op_map *op, -u8 tile_mask,bool read_only,bool is_null) +u8 tile_mask,bool read_only,bool is_null, +u32 pat_index) {struct xe_bo *bo = op->gem.obj ?gem_to_xe_bo (op->gem.obj) :NULL ;struct xe_vma *vma;@@ -2425,7 +2428,7 @@ static struct xe_vma *new_vma(struct xe_vm *vm, struct drm_gpuva_op_map *op, vma =xe_vma_create (vm, bo, op->gem.offset, op->va.addr, op->va.addr + op->va.range -1 , read_only, is_null, - tile_mask); + tile_mask, pat_index);if (bo)xe_bo_unlock (bo, &ww);@@ -2539,7 +2542,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, &op->base.map, op->tile_mask, op->map.read_only, - op->map.is_null); + op->map.is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2567,7 +2570,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, op->base.remap.prev, op->tile_mask, read_only, - is_null); + is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2603,7 +2606,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, op->base.remap.next, op->tile_mask, read_only, - is_null); + is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -3158,8 +3161,14 @@ static int vm_bind_ioctl_check_args(struct xe_device *xe,u32 obj = (*bind_ops)[i].obj;u64 obj_offset = (*bind_ops)[i].obj_offset;u32 region = (*bind_ops)[i].region; +u32 pat_index = (*bind_ops)[i].pat_index;bool is_null = op &XE_VM_BIND_FLAG_NULL ; +if (XE_IOCTL_DBG (xe, pat_index >= xe->info.pat_table_n_entries)) { + err = -EINVAL ; + goto free_bind_ops; + } +if (i ==0 ) { *async = !!(op &XE_VM_BIND_FLAG_ASYNC ); } else if (XE_IOCTL_DBG(xe, !*async) ||@@ -3346,8 +3355,25 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)struct drm_gem_object *gem_obj;u64 range = bind_ops[i].range;u64 addr = bind_ops[i].addr; +u32 op = bind_ops[i].op;u32 obj = bind_ops[i].obj;u64 obj_offset = bind_ops[i].obj_offset; +u32 pat_index = bind_ops[i].pat_index; +u16 coh_mode; + + pat_index =array_index_nospec (pat_index, + xe->info.pat_table_n_entries); + coh_mode =xe_pat_index_get_coh_mode (xe, pat_index); +if (XE_IOCTL_DBG (xe, !coh_mode)) { + err = -EINVAL ; + goto put_obj; + } + +if (XE_IOCTL_DBG (xe,VM_BIND_OP (op) ==XE_VM_BIND_OP_MAP_USERPTR && + coh_mode ==XE_GEM_COHERENCY_NONE )) { + err = -EINVAL ; + goto put_obj; + }if (!obj)continue ;@@ -3375,6 +3401,22 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) goto put_obj; } } + +if (bos[i]->coh_mode) { +if (XE_IOCTL_DBG (xe, bos[i]->coh_mode != coh_mode)) { + err = -EINVAL ; + goto put_obj; + } + }else if (XE_IOCTL_DBG (xe, coh_mode ==XE_GEM_COHERENCY_NONE )) { +/* +* Imported dma-buf from a different device should +* require 1way or 2way coherency since we don't know +* how it was mapped on CPU. Just assume is it +* potentially cached on CPU side. +*/ + err = -EINVAL ; + goto put_obj; + } }if (args->num_syncs) {@@ -3412,10 +3454,11 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)u64 obj_offset = bind_ops[i].obj_offset;u8 tile_mask = bind_ops[i].tile_mask;u32 region = bind_ops[i].region; +u32 pat_index = bind_ops[i].pat_index; ops[i] =vm_bind_ioctl_ops_create (vm, bos[i], obj_offset, addr, range, op, tile_mask, - region); + region, pat_index);if (IS_ERR (ops[i])) { err =PTR_ERR (ops[i]); ops[i] =NULL ;drivers/gpu/drm/xe/xe_vm_types.h +9
-4@@ -111,6 +111,11 @@ struct xe_vma {*/ u8 tile_present; +/** +* @pat_index: The pat index to use when encoding the PTEs for this vma. +*/ +u32 pat_index; +struct {struct list_head rebind_link; } notifier;@@ -338,10 +343,8 @@ struct xe_vm {bool batch_invalidate_tlb;struct { -u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache); -u64 (*pde_encode)(struct xe_device *xe,u64 pde_pat, -enum xe_cache_level cache); +u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat,u32 pat_index); +u64 (*pde_encode)(struct xe_device *xe,u64 pde_pat,u32 pat_index); } pat_encode; };@@ -417,6 +420,8 @@ struct xe_vma_op {struct async_op_fence *fence;/** @tile_mask: gt mask for this operation */ u8 tile_mask; +/** @pat_index: The pat index to use for this operation. */ +u32 pat_index;/** @flags: operation flags */ enum xe_vma_op_flags flags;include/uapi/drm/xe_drm.h +39
-2@@ -600,8 +600,45 @@ struct drm_xe_vm_bind_op {*/ __u32 obj; -/** @pad: MBZ */ -__u32 pad; +/** +* @pat_index: The platform defined @pat_index to use for this mapping. +* The index basically maps to some predefined memory attributes, +* including things like caching, coherency and likely other stuff in +* the future. The exact meaning of the pat_index is platform specific +* and defined in the Bspec and PRMs. When the KMD sets up the binding +* the index here is encoded into the ppGTT PTE. +* +* For coherency the @pat_index needs to match the +* drm_xe_gem_create.coh_mode, so either XE_GEM_COHERENCY_NONE, +* XE_GEM_COHERENCY_1WAY or XE_GEM_COHERENCY_2WAY. The KMD will extract +* the coherency mode from the @pat_index and reject if there is a +* mismatch (see note below for pre-MTL platforms). +* +* Note: On pre-MTL platforms there is only a caching mode and no +* explicit coherency mode, but on such hardware there is always a +* shared-LLC (or is dgpu) so all GT memory accesses are coherent with +* CPU caches even with the caching mode set as uncached. It's only the +* display engine that is incoherent (on dgpu it must be in VRAM which +* is always mapped as WC on the CPU). However to keep the uapi somewhat +* consistent with newer platforms the KMD groups the different cache +* levels into the following coherency buckets on all pre-MTL platforms: +* +* ppGTT UC -> XE_GEM_COHERENCY_NONE +* ppGTT WC -> XE_GEM_COHERENCY_NONE +* ppGTT WT -> XE_GEM_COHERENCY_NONE +* ppGTT WB -> XE_GEM_COHERENCY_2WAY +* +* In practice UC/WC/WT should only ever used for scanout surfaces on +* such platforms since it is only the display engine that is actually +* incoherent. Everything else should typically use WB given that we +* have a shared-LLC. On MTL+ this completely changes (also potentially +* no shared-LLC) and the HW defines the coherency mode as part of the +* @pat_index. +* +* Note: For userptr and externally imported dma-buf the kernel expects +* either 1WAY or 2WAY for the @pat_index. +*/ +__u32 pat_index;union {/**
Re: [Intel-xe] [RFC 5/5] drm/xe/uapi: support pat_index selection with vm_bind
Quoted 27 lines · Jose
Quoted 18 lines · Matthew Auld
Allow userspace to directly control the
pat_indexfor a given vm binding. This should allow directly controlling the coherency, caching and potentially other stuff in the future for the ppGTT binding.The exact meaning behind the
pat_indexis very platform specific (see BSpec or PRMs) but effectively maps to some predefined memory attributes. From the KMD pov we only care about the coherency that is provided by thepat_index, which falls into either NONE, 1WAY or 2WAY. Thevm_bindcoherency mode for the givenpat_indexneeds to match the givencoh_modethat was set at object creation. For platforms that lack the explicit coherency mode, we treat UC/WT/WC as NONE and WB as 2WAY.For userptr mappings we lack a corresponding gem object, so the expected coherency mode is instead implicit and must fall into either 1WAY or 2WAY. Trying to use NONE will be rejected by the kernel. For imported dma-buf (from a different device) the coherency mode is also implicit and must also be either 1WAY or 2WAY.
Quoted 7 lines
As part of adding
pat_indexsupport withvm_bindwe also need stop usingxe_cache_leveland instead use thepat_indexin various places. We still make use ofxe_cache_level, but only as a convenience for kernel internal objectsi (internally it maps to some reasonablepat_index). For now this is just a 1:1 conversion of the existing code, however for platforms like MTL+ we might need to give more control throughbo_createor stop using WB on the CPU side if we need CPU access.
Quoted 1 line
xe_ttm_tt_create()caching should not also take into consideration the PAT index?
Which PAT index? At least for userspace objects we might not know the pat_index at this stage, also the pat_index is more per vma, while the CPU caching mode in xe_ttm_tt_create() is per object. At gem create we only need to consider the overall coh_mode of every pat_index that will map the object to determine if the CPU caching mode is "valid".
Quoted 609 lines
Quoted 608 lines
Bspec: 45101, 44235 #xe
Bspec: 70552, 71582, 59400 #xe2Signed-off-by Matthew Auld <[email protected]>Cc Pallavi Mishra <[email protected]>Cc Thomas Hellström <[email protected]>Cc Joonas Lahtinen <[email protected]>Cc Lucas De Marchi <[email protected]>Cc Matt Roper <[email protected]>Cc José Roberto de Souza <[email protected]>Cc Filip Hazubski <[email protected]>Cc Carl Zhang <[email protected]>Cc Effie Yu <[email protected]>--- drivers/gpu/drm/i915/gt/intel_gtt.h | 2 +- drivers/gpu/drm/xe/tests/xe_migrate.c | 2 +- drivers/gpu/drm/xe/xe_ggtt.c | 7 ++- drivers/gpu/drm/xe/xe_ggtt_types.h | 2 +- drivers/gpu/drm/xe/xe_migrate.c | 14 ++--- drivers/gpu/drm/xe/xe_pt.c | 32 +++++------- drivers/gpu/drm/xe/xe_pt.h | 6 +-- drivers/gpu/drm/xe/xe_vm.c | 73 +++++++++++++++++++++------ drivers/gpu/drm/xe/xe_vm_types.h | 13 +++-- include/uapi/drm/xe_drm.h | 41 ++++++++++++++- 10 files changed, 134 insertions(+), 58 deletions(-)drivers/gpu/drm/i915/gt/intel_gtt.h +1
-1@@ -302,7 +302,7 @@ struct i915_address_space { (*alloc_scratch_dma)(struct i915_address_space *vm, int sz);u64 (*pte_encode)(dma_addr_t addr, -unsigned int pat_index, +u32 pat_index,u32 flags);/* Create a valid PTE */ #define PTE_READ_ONLY BIT(0)#define PTE_LM BIT(1)drivers/gpu/drm/xe/tests/xe_migrate.c +1
-1@@ -301,7 +301,7 @@ static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test)/* First part of the test, are we updating our pagetable bo with a new entry? */ xe_map_wr (xe, &bo->vmap,XE_PAGE_SIZE * (NUM_KERNEL_PDE -1 ), u64,0xdeaddeadbeefbeef ); - expected =xe_pte_encode (pt,0 ,XE_CACHE_WB ,0 ); + expected =xe_pte_encode (pt,0 ,xe_pat_get_index (xe,XE_CACHE_WB ),0 );if (m->q->vm->flags &XE_VM_FLAG_64K ) expected |=XE_PTE_PS64 ;if (xe_bo_is_vram (pt))drivers/gpu/drm/xe/xe_ggtt.c +3
-4@@ -41,7 +41,8 @@ u64 xe_ggtt_pte_encode(struct xe_bo *bo, u64 bo_offset) pte |=XE_GGTT_PTE_DM ;if ((ggtt->pat_encode).pte_encode) - pte = (ggtt->pat_encode).pte_encode (xe, pte,XE_CACHE_WB_1_WAY ); + pte = (ggtt->pat_encode).pte_encode (xe, pte, +xe_pat_get_index (xe,XE_CACHE_WB_1_WAY ));return pte; }@@ -102,10 +103,8 @@ static void primelockdep(struct xe_ggtt *ggtt) }static u64 xelpg_ggtt_pte_encode_pat (struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELPG_GGTT_PTE_PAT_MASK );if (pat_index &BIT (0 ))drivers/gpu/drm/xe/xe_ggtt_types.h +1
-1@@ -31,7 +31,7 @@ struct xe_ggtt {struct {u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache); +u32 pat_index); } pat_encode; };drivers/gpu/drm/xe/xe_migrate.c +8
-6@@ -24,6 +24,7 @@#include "xe_lrc.h" #include "xe_map.h" #include "xe_mocs.h" +#include "xe_pat.h" #include "xe_pt.h" #include "xe_res_cursor.h" #include "xe_sched_job.h" @@ -162,6 +163,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,u32 num_entries =NUM_PT_SLOTS , num_level = vm->pt_root[id]->level;u32 map_ofs, level, i;struct xe_bo *bo, *batch = tile->mem.kernel_bb_pool->bo; +u32 pat_index =xe_pat_get_index (xe,XE_CACHE_WB );u64 entry;int ret;@@ -189,14 +191,14 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,return ret; } - entry =xe_pde_encode (bo, bo->size -XE_PAGE_SIZE ,XE_CACHE_WB ); + entry =xe_pde_encode (bo, bo->size -XE_PAGE_SIZE , pat_index);xe_pt_write (xe, &vm->pt_root[id]->bo->vmap,0 , entry); map_ofs = (num_entries - num_level) *XE_PAGE_SIZE ;/* Map the entire BO in our level 0 pt */ for (i =0 , level =0 ; i < num_entries; level++) { - entry =xe_pte_encode (bo, i *XE_PAGE_SIZE ,XE_CACHE_WB ,0 ); + entry =xe_pte_encode (bo, i *XE_PAGE_SIZE , pat_index,0 );xe_map_wr (xe, &bo->vmap, map_ofs + level *8 , u64, entry);@@ -214,7 +216,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,for (i =0 ; i < batch->size; i += vm->flags &XE_VM_FLAG_64K ?XE_64K_PAGE_SIZE :XE_PAGE_SIZE ) { - entry =xe_pte_encode (batch, i,XE_CACHE_WB ,0 ); + entry =xe_pte_encode (batch, i, pat_index,0 );xe_map_wr (xe, &bo->vmap, map_ofs + level *8 , u64, entry);@@ -239,7 +241,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m, flags =XE_PDE_64K ; entry =xe_pde_encode (bo, map_ofs + (level -1 ) * -XE_PAGE_SIZE ,XE_CACHE_WB ); +XE_PAGE_SIZE , pat_index);xe_map_wr (xe, &bo->vmap, map_ofs +XE_PAGE_SIZE * level, u64, entry | flags); }@@ -247,7 +249,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,/* Write PDE's that point to our BO. */ for (i =0 ; i < num_entries - num_level; i++) { entry =xe_pde_encode (bo, i *XE_PAGE_SIZE , -XE_CACHE_WB ); + pat_index);xe_map_wr (xe, &bo->vmap, map_ofs +XE_PAGE_SIZE + (i +1 ) *8 , u64, entry);@@ -1256,7 +1258,7 @@ xe_migrate_update_pgtables(struct xe_migrate *m,XE_WARN_ON (pt_bo->size !=SZ_4K ); - addr =xe_pte_encode (pt_bo,0 ,XE_CACHE_WB ,0 ); + addr =xe_pte_encode (pt_bo,0 ,xe_pat_get_index (xe,XE_CACHE_WB ),0 ); bb->cs[bb->len++] =lower_32_bits (addr); bb->cs[bb->len++] =upper_32_bits (addr); }drivers/gpu/drm/xe/xe_pt.c +12
-20@@ -10,6 +10,7 @@#include "xe_gt.h" #include "xe_gt_tlb_invalidation.h" #include "xe_migrate.h" +#include "xe_pat.h" #include "xe_pt_types.h" #include "xe_pt_walk.h" #include "xe_res_cursor.h" @@ -57,24 +58,22 @@ static struct xe_pt *xe_pt_entry(struct xe_pt_dir *pt_dir, unsigned int index)* * Return: An encoded page directory entry. No errors. */ -u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level cache) +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 pat_index) {u64 pde;struct xe_vm *vm = bo->vm;struct xe_device *xe = vm->xe; - pde =xe_bo_addr (bo, bo_offset,XE_PAGE_SIZE ); pde |=XE_PAGE_PRESENT |XE_PAGE_RW ;if ((vm->pat_encode).pde_encode) - pde = (vm->pat_encode).pde_encode (xe, pde, cache); + pde = (vm->pat_encode).pde_encode (xe, pde, pat_index);return pde; } -static u64 __pte_encode (u64 pte,enum xe_cache_level cache, +static u64 __pte_encode (u64 pte,u32 pat_index,struct xe_vma *vma,u32 pt_level) {struct xe_vm *vm =xe_vma_vm (vma);@@ -89,7 +88,7 @@ static u64 __pte_encode(u64 pte, enum xe_cache_level cache, pte |=XE_PTE_NULL ;if ((vm->pat_encode).pte_encode) - pte = (vm->pat_encode).pte_encode (xe, pte, cache); + pte = (vm->pat_encode).pte_encode (xe, pte, pat_index);if (pt_level ==1 ) pte |=XE_PDE_PS_2M ;@@ -112,7 +111,7 @@ static u64 __pte_encode(u64 pte, enum xe_cache_level cache,* * Return: An encoded page-table entry. No errors. */ -u64 xe_pte_encode (struct xe_bo *bo,u64 offset,enum xe_cache_level cache, +u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 pat_index,u32 pt_level) {u64 pte;@@ -121,7 +120,7 @@ u64 xe_pte_encode(struct xe_bo *bo, u64 offset, enum xe_cache_level cache,if (xe_bo_is_vram (bo) ||xe_bo_is_stolen_devmem (bo)) pte |=XE_PPGTT_PTE_DM ; -return __pte_encode (pte, cache,NULL , pt_level); +return __pte_encode (pte, pat_index,NULL , pt_level); } static u64 __xe_pt_empty_pte(struct xe_tile *tile, struct xe_vm *vm,@@ -134,12 +133,12 @@ static u64 __xe_pt_empty_pte(struct xe_tile *tile, struct xe_vm *vm,if (level ==0 ) {u64 empty =xe_pte_encode (vm->scratch_bo[id],0 , -XE_CACHE_WB ,0 ); +xe_pat_get_index (vm->xe,XE_CACHE_WB ),0 );return empty; }else {return xe_pde_encode (vm->scratch_pt[id][level -1 ]->bo,0 , -XE_CACHE_WB ); +xe_pat_get_index (vm->xe,XE_CACHE_WB )); } }@@ -368,8 +367,6 @@ struct xe_pt_stage_bind_walk {struct xe_vm *vm;/** @tile: The tile we're building for. */ struct xe_tile *tile; -/** @cache: Desired cache level for the ptes */ -enum xe_cache_level cache;/** @default_pte: PTE flag only template. No address is associated */ u64 default_pte;/** @dma_offset: DMA offset to add to the PTE. */ @@ -604,7 +601,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset, pte =__pte_encode (is_null ?0 :xe_res_dma (curs) + xe_walk->dma_offset, - xe_walk->cache, xe_walk->vma, level); + xe_walk->vma->pat_index, xe_walk->vma, level); pte |= xe_walk->default_pte;/* @@ -669,7 +666,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset, xe_child->is_compact = true; } - pte =xe_pde_encode (xe_child->bo,0 , xe_walk->cache) | flags; + pte =xe_pde_encode (xe_child->bo,0 , xe_walk->vma->pat_index) | flags; ret =xe_pt_insert_entry (xe_walk, xe_parent, offset, xe_child, pte); }@@ -730,13 +727,8 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma,if (vma && vma->gpuva.flags &XE_VMA_ATOMIC_PTE_BIT ) xe_walk.default_pte |=XE_USM_PPGTT_PTE_AE ; xe_walk.dma_offset =vram_region_gpu_offset (bo->ttm.resource); - xe_walk.cache =XE_CACHE_WB ; - }else { -if (!xe_vma_has_no_bo (vma) && bo->flags &XE_BO_SCANOUT_BIT ) - xe_walk.cache =XE_CACHE_WT ; -else - xe_walk.cache =XE_CACHE_WB ; } +if (!xe_vma_has_no_bo (vma) &&xe_bo_is_stolen (bo)) xe_walk.dma_offset =xe_ttm_stolen_gpu_offset (xe_bo_device (bo));drivers/gpu/drm/xe/xe_pt.h +2
-4@@ -45,10 +45,8 @@ __xe_pt_unbind_vma(struct xe_tile *tile, struct xe_vma *vma, struct xe_exec_queubool xe_pt_zap_ptes (struct xe_tile *tile,struct xe_vma *vma); -u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level level); +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 pat_index); -u64 xe_pte_encode (struct xe_bo *bo,u64 offset,enum xe_cache_level cache, -u32 pt_level); +u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 pat_index,u32 pt_level);#endif drivers/gpu/drm/xe/xe_vm.c +58
-15@@ -6,6 +6,7 @@#include "xe_vm.h" #include <linux/dma-fence-array.h> +#include <linux/nospec.h> #include <drm/drm_print.h> #include <drm/ttm/ttm_execbuf_util.h> @@ -874,7 +875,8 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, u64 start, u64 end, bool read_only, bool is_null, - u8 tile_mask) + u8 tile_mask, + u32 pat_index) { struct xe_vma *vma;struct xe_tile *tile;@@ -913,6 +915,8 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, vma->tile_mask |=0x1 << id; } + vma->pat_index = pat_index; +if (vm->xe->info.platform ==XE_PVC ) vma->gpuva.flags |=XE_VMA_ATOMIC_PTE_BIT ;@@ -1194,10 +1198,8 @@ static void xe_vma_op_work_func(struct work_struct *w);static void vm_destroy_work_func (struct work_struct *w);static u64 xelp_ppgtt_pde_encode_pat (struct xe_device *xe,u64 pde_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pde_pat &= ~(XELP_PDE_PAT_MASK );if (pat_index &BIT (0 ))@@ -1213,10 +1215,8 @@ static u64 xelp_ppgtt_pde_encode_pat(struct xe_device *xe, u64 pde_pat, }static u64 xelp_ppgtt_pte_encode_pat (struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELP_PTE_PAT_MASK );if (pat_index &BIT (0 ))@@ -1622,7 +1622,7 @@ struct xe_vm *xe_vm_lookup(struct xe_file *xef, u32 id)u64 xe_vm_pdp4_descriptor (struct xe_vm *vm,struct xe_tile *tile) {return xe_pde_encode (vm->pt_root[tile->id]->bo,0 , -XE_CACHE_WB ); +xe_pat_get_index (vm->xe,XE_CACHE_WB )); } static struct dma_fence *@@ -2311,7 +2311,7 @@ static void print_op(struct xe_device *xe, struct drm_gpuva_op *op)static struct drm_gpuva_ops *vm_bind_ioctl_ops_create (struct xe_vm *vm,struct xe_bo *bo,u64 bo_offset_or_userptr,u64 addr,u64 range, -u32 operation,u8 tile_mask,u32 region) +u32 operation,u8 tile_mask,u32 region,u32 pat_index) {struct drm_gem_object *obj = bo ? &bo->ttm.base :NULL ;struct ww_acquire_ctx ww;@@ -2339,6 +2339,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo,struct xe_vma_op *op =gpuva_op_to_vma_op (__op); op->tile_mask = tile_mask; + op->pat_index = pat_index; op->map.immediate = operation &XE_VM_BIND_FLAG_IMMEDIATE ; op->map.read_only =@@ -2366,6 +2367,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo,struct xe_vma_op *op =gpuva_op_to_vma_op (__op); op->tile_mask = tile_mask; + op->pat_index = pat_index; op->prefetch.region = region; }break ;@@ -2408,7 +2410,8 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo, }static struct xe_vma *new_vma (struct xe_vm *vm,struct drm_gpuva_op_map *op, -u8 tile_mask,bool read_only,bool is_null) +u8 tile_mask,bool read_only,bool is_null, +u32 pat_index) {struct xe_bo *bo = op->gem.obj ?gem_to_xe_bo (op->gem.obj) :NULL ;struct xe_vma *vma;@@ -2425,7 +2428,7 @@ static struct xe_vma *new_vma(struct xe_vm *vm, struct drm_gpuva_op_map *op, vma =xe_vma_create (vm, bo, op->gem.offset, op->va.addr, op->va.addr + op->va.range -1 , read_only, is_null, - tile_mask); + tile_mask, pat_index);if (bo)xe_bo_unlock (bo, &ww);@@ -2539,7 +2542,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, &op->base.map, op->tile_mask, op->map.read_only, - op->map.is_null); + op->map.is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2567,7 +2570,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, op->base.remap.prev, op->tile_mask, read_only, - is_null); + is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2603,7 +2606,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, op->base.remap.next, op->tile_mask, read_only, - is_null); + is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -3158,8 +3161,14 @@ static int vm_bind_ioctl_check_args(struct xe_device *xe,u32 obj = (*bind_ops)[i].obj;u64 obj_offset = (*bind_ops)[i].obj_offset;u32 region = (*bind_ops)[i].region; +u32 pat_index = (*bind_ops)[i].pat_index;bool is_null = op &XE_VM_BIND_FLAG_NULL ; +if (XE_IOCTL_DBG (xe, pat_index >= xe->info.pat_table_n_entries)) { + err = -EINVAL ; + goto free_bind_ops; + } +if (i ==0 ) { *async = !!(op &XE_VM_BIND_FLAG_ASYNC ); } else if (XE_IOCTL_DBG(xe, !*async) ||@@ -3346,8 +3355,25 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)struct drm_gem_object *gem_obj;u64 range = bind_ops[i].range;u64 addr = bind_ops[i].addr; +u32 op = bind_ops[i].op;u32 obj = bind_ops[i].obj;u64 obj_offset = bind_ops[i].obj_offset; +u32 pat_index = bind_ops[i].pat_index; +u16 coh_mode; + + pat_index =array_index_nospec (pat_index, + xe->info.pat_table_n_entries); + coh_mode =xe_pat_index_get_coh_mode (xe, pat_index); +if (XE_IOCTL_DBG (xe, !coh_mode)) { + err = -EINVAL ; + goto put_obj; + } + +if (XE_IOCTL_DBG (xe,VM_BIND_OP (op) ==XE_VM_BIND_OP_MAP_USERPTR && + coh_mode ==XE_GEM_COHERENCY_NONE )) { + err = -EINVAL ; + goto put_obj; + }if (!obj)continue ;@@ -3375,6 +3401,22 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) goto put_obj; } } + +if (bos[i]->coh_mode) { +if (XE_IOCTL_DBG (xe, bos[i]->coh_mode != coh_mode)) { + err = -EINVAL ; + goto put_obj; + } + }else if (XE_IOCTL_DBG (xe, coh_mode ==XE_GEM_COHERENCY_NONE )) { +/* +* Imported dma-buf from a different device should +* require 1way or 2way coherency since we don't know +* how it was mapped on CPU. Just assume is it +* potentially cached on CPU side. +*/ + err = -EINVAL ; + goto put_obj; + } }if (args->num_syncs) {@@ -3412,10 +3454,11 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)u64 obj_offset = bind_ops[i].obj_offset;u8 tile_mask = bind_ops[i].tile_mask;u32 region = bind_ops[i].region; +u32 pat_index = bind_ops[i].pat_index; ops[i] =vm_bind_ioctl_ops_create (vm, bos[i], obj_offset, addr, range, op, tile_mask, - region); + region, pat_index);if (IS_ERR (ops[i])) { err =PTR_ERR (ops[i]); ops[i] =NULL ;drivers/gpu/drm/xe/xe_vm_types.h +9
-4@@ -111,6 +111,11 @@ struct xe_vma {*/ u8 tile_present; +/** +* @pat_index: The pat index to use when encoding the PTEs for this vma. +*/ +u32 pat_index; +struct {struct list_head rebind_link; } notifier;@@ -338,10 +343,8 @@ struct xe_vm {bool batch_invalidate_tlb;struct { -u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache); -u64 (*pde_encode)(struct xe_device *xe,u64 pde_pat, -enum xe_cache_level cache); +u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat,u32 pat_index); +u64 (*pde_encode)(struct xe_device *xe,u64 pde_pat,u32 pat_index); } pat_encode; };@@ -417,6 +420,8 @@ struct xe_vma_op {struct async_op_fence *fence;/** @tile_mask: gt mask for this operation */ u8 tile_mask; +/** @pat_index: The pat index to use for this operation. */ +u32 pat_index;/** @flags: operation flags */ enum xe_vma_op_flags flags;include/uapi/drm/xe_drm.h +39
-2@@ -600,8 +600,45 @@ struct drm_xe_vm_bind_op {*/ __u32 obj; -/** @pad: MBZ */ -__u32 pad; +/** +* @pat_index: The platform defined @pat_index to use for this mapping. +* The index basically maps to some predefined memory attributes, +* including things like caching, coherency and likely other stuff in +* the future. The exact meaning of the pat_index is platform specific +* and defined in the Bspec and PRMs. When the KMD sets up the binding +* the index here is encoded into the ppGTT PTE. +* +* For coherency the @pat_index needs to match the +* drm_xe_gem_create.coh_mode, so either XE_GEM_COHERENCY_NONE, +* XE_GEM_COHERENCY_1WAY or XE_GEM_COHERENCY_2WAY. The KMD will extract +* the coherency mode from the @pat_index and reject if there is a +* mismatch (see note below for pre-MTL platforms). +* +* Note: On pre-MTL platforms there is only a caching mode and no +* explicit coherency mode, but on such hardware there is always a +* shared-LLC (or is dgpu) so all GT memory accesses are coherent with +* CPU caches even with the caching mode set as uncached. It's only the +* display engine that is incoherent (on dgpu it must be in VRAM which +* is always mapped as WC on the CPU). However to keep the uapi somewhat +* consistent with newer platforms the KMD groups the different cache +* levels into the following coherency buckets on all pre-MTL platforms: +* +* ppGTT UC -> XE_GEM_COHERENCY_NONE +* ppGTT WC -> XE_GEM_COHERENCY_NONE +* ppGTT WT -> XE_GEM_COHERENCY_NONE +* ppGTT WB -> XE_GEM_COHERENCY_2WAY +* +* In practice UC/WC/WT should only ever used for scanout surfaces on +* such platforms since it is only the display engine that is actually +* incoherent. Everything else should typically use WB given that we +* have a shared-LLC. On MTL+ this completely changes (also potentially +* no shared-LLC) and the HW defines the coherency mode as part of the +* @pat_index. +* +* Note: For userptr and externally imported dma-buf the kernel expects +* either 1WAY or 2WAY for the @pat_index. +*/ +__u32 pat_index;union {/**
Re: [Intel-xe] [RFC 5/5] drm/xe/uapi: support pat_index selection with vm_bind
Quoted 28 lines · Matthew Auld
Quoted 27 lines · Jose
Quoted 25 lines · Matthew Auld
Allow userspace to directly control the
pat_indexfor a given vm binding. This should allow directly controlling the coherency, caching and potentially other stuff in the future for the ppGTT binding.The exact meaning behind the
pat_indexis very platform specific (see BSpec or PRMs) but effectively maps to some predefined memory attributes. From the KMD pov we only care about the coherency that is provided by thepat_index, which falls into either NONE, 1WAY or 2WAY. Thevm_bindcoherency mode for the givenpat_indexneeds to match the givencoh_modethat was set at object creation. For platforms that lack the explicit coherency mode, we treat UC/WT/WC as NONE and WB as 2WAY.For userptr mappings we lack a corresponding gem object, so the expected coherency mode is instead implicit and must fall into either 1WAY or 2WAY. Trying to use NONE will be rejected by the kernel. For imported dma-buf (from a different device) the coherency mode is also implicit and must also be either 1WAY or 2WAY.
As part of adding
pat_indexsupport withvm_bindwe also need stop usingxe_cache_leveland instead use thepat_indexin various places. We still make use ofxe_cache_level, but only as a convenience for kernel internal objectsi (internally it maps to some reasonablepat_index). For now this is just a 1:1 conversion of the existing code, however for platforms like MTL+ we might need to give more control throughbo_createor stop using WB on the CPU side if we need CPU access.
Quoted 7 lines
Quoted 1 line
xe_ttm_tt_create()caching should not also take into consideration the PAT index?Which PAT index? At least for userspace objects we might not know the
pat_indexat this stage, also thepat_indexis more per vma, while the CPU caching mode inxe_ttm_tt_create()is per object. At gem create we only need to consider the overallcoh_modeof everypat_indexthat will map the object to determine if the CPU caching mode is "valid".
Yeah it is too late.
I should not consider smem_caching in drm_xe_gem_create? It that what will be used as caching mode for mmap?
Quoted 610 lines
Quoted 610 lines
Quoted 609 lines
Bspec: 45101, 44235 #xe
Bspec: 70552, 71582, 59400 #xe2Signed-off-by Matthew Auld <[email protected]>Cc Pallavi Mishra <[email protected]>Cc Thomas Hellström <[email protected]>Cc Joonas Lahtinen <[email protected]>Cc Lucas De Marchi <[email protected]>Cc Matt Roper <[email protected]>Cc José Roberto de Souza <[email protected]>Cc Filip Hazubski <[email protected]>Cc Carl Zhang <[email protected]>Cc Effie Yu <[email protected]>--- drivers/gpu/drm/i915/gt/intel_gtt.h | 2 +- drivers/gpu/drm/xe/tests/xe_migrate.c | 2 +- drivers/gpu/drm/xe/xe_ggtt.c | 7 ++- drivers/gpu/drm/xe/xe_ggtt_types.h | 2 +- drivers/gpu/drm/xe/xe_migrate.c | 14 ++--- drivers/gpu/drm/xe/xe_pt.c | 32 +++++------- drivers/gpu/drm/xe/xe_pt.h | 6 +-- drivers/gpu/drm/xe/xe_vm.c | 73 +++++++++++++++++++++------ drivers/gpu/drm/xe/xe_vm_types.h | 13 +++-- include/uapi/drm/xe_drm.h | 41 ++++++++++++++- 10 files changed, 134 insertions(+), 58 deletions(-)drivers/gpu/drm/i915/gt/intel_gtt.h +1
-1@@ -302,7 +302,7 @@ struct i915_address_space { (*alloc_scratch_dma)(struct i915_address_space *vm, int sz);u64 (*pte_encode)(dma_addr_t addr, -unsigned int pat_index, +u32 pat_index,u32 flags);/* Create a valid PTE */ #define PTE_READ_ONLY BIT(0)#define PTE_LM BIT(1)drivers/gpu/drm/xe/tests/xe_migrate.c +1
-1@@ -301,7 +301,7 @@ static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test)/* First part of the test, are we updating our pagetable bo with a new entry? */ xe_map_wr (xe, &bo->vmap,XE_PAGE_SIZE * (NUM_KERNEL_PDE -1 ), u64,0xdeaddeadbeefbeef ); - expected =xe_pte_encode (pt,0 ,XE_CACHE_WB ,0 ); + expected =xe_pte_encode (pt,0 ,xe_pat_get_index (xe,XE_CACHE_WB ),0 );if (m->q->vm->flags &XE_VM_FLAG_64K ) expected |=XE_PTE_PS64 ;if (xe_bo_is_vram (pt))drivers/gpu/drm/xe/xe_ggtt.c +3
-4@@ -41,7 +41,8 @@ u64 xe_ggtt_pte_encode(struct xe_bo *bo, u64 bo_offset) pte |=XE_GGTT_PTE_DM ;if ((ggtt->pat_encode).pte_encode) - pte = (ggtt->pat_encode).pte_encode (xe, pte,XE_CACHE_WB_1_WAY ); + pte = (ggtt->pat_encode).pte_encode (xe, pte, +xe_pat_get_index (xe,XE_CACHE_WB_1_WAY ));return pte; }@@ -102,10 +103,8 @@ static void primelockdep(struct xe_ggtt *ggtt) }static u64 xelpg_ggtt_pte_encode_pat (struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELPG_GGTT_PTE_PAT_MASK );if (pat_index &BIT (0 ))drivers/gpu/drm/xe/xe_ggtt_types.h +1
-1@@ -31,7 +31,7 @@ struct xe_ggtt {struct {u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache); +u32 pat_index); } pat_encode; };drivers/gpu/drm/xe/xe_migrate.c +8
-6@@ -24,6 +24,7 @@#include "xe_lrc.h" #include "xe_map.h" #include "xe_mocs.h" +#include "xe_pat.h" #include "xe_pt.h" #include "xe_res_cursor.h" #include "xe_sched_job.h" @@ -162,6 +163,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,u32 num_entries =NUM_PT_SLOTS , num_level = vm->pt_root[id]->level;u32 map_ofs, level, i;struct xe_bo *bo, *batch = tile->mem.kernel_bb_pool->bo; +u32 pat_index =xe_pat_get_index (xe,XE_CACHE_WB );u64 entry;int ret;@@ -189,14 +191,14 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,return ret; } - entry =xe_pde_encode (bo, bo->size -XE_PAGE_SIZE ,XE_CACHE_WB ); + entry =xe_pde_encode (bo, bo->size -XE_PAGE_SIZE , pat_index);xe_pt_write (xe, &vm->pt_root[id]->bo->vmap,0 , entry); map_ofs = (num_entries - num_level) *XE_PAGE_SIZE ;/* Map the entire BO in our level 0 pt */ for (i =0 , level =0 ; i < num_entries; level++) { - entry =xe_pte_encode (bo, i *XE_PAGE_SIZE ,XE_CACHE_WB ,0 ); + entry =xe_pte_encode (bo, i *XE_PAGE_SIZE , pat_index,0 );xe_map_wr (xe, &bo->vmap, map_ofs + level *8 , u64, entry);@@ -214,7 +216,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,for (i =0 ; i < batch->size; i += vm->flags &XE_VM_FLAG_64K ?XE_64K_PAGE_SIZE :XE_PAGE_SIZE ) { - entry =xe_pte_encode (batch, i,XE_CACHE_WB ,0 ); + entry =xe_pte_encode (batch, i, pat_index,0 );xe_map_wr (xe, &bo->vmap, map_ofs + level *8 , u64, entry);@@ -239,7 +241,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m, flags =XE_PDE_64K ; entry =xe_pde_encode (bo, map_ofs + (level -1 ) * -XE_PAGE_SIZE ,XE_CACHE_WB ); +XE_PAGE_SIZE , pat_index);xe_map_wr (xe, &bo->vmap, map_ofs +XE_PAGE_SIZE * level, u64, entry | flags); }@@ -247,7 +249,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,/* Write PDE's that point to our BO. */ for (i =0 ; i < num_entries - num_level; i++) { entry =xe_pde_encode (bo, i *XE_PAGE_SIZE , -XE_CACHE_WB ); + pat_index);xe_map_wr (xe, &bo->vmap, map_ofs +XE_PAGE_SIZE + (i +1 ) *8 , u64, entry);@@ -1256,7 +1258,7 @@ xe_migrate_update_pgtables(struct xe_migrate *m,XE_WARN_ON (pt_bo->size !=SZ_4K ); - addr =xe_pte_encode (pt_bo,0 ,XE_CACHE_WB ,0 ); + addr =xe_pte_encode (pt_bo,0 ,xe_pat_get_index (xe,XE_CACHE_WB ),0 ); bb->cs[bb->len++] =lower_32_bits (addr); bb->cs[bb->len++] =upper_32_bits (addr); }drivers/gpu/drm/xe/xe_pt.c +12
-20@@ -10,6 +10,7 @@#include "xe_gt.h" #include "xe_gt_tlb_invalidation.h" #include "xe_migrate.h" +#include "xe_pat.h" #include "xe_pt_types.h" #include "xe_pt_walk.h" #include "xe_res_cursor.h" @@ -57,24 +58,22 @@ static struct xe_pt *xe_pt_entry(struct xe_pt_dir *pt_dir, unsigned int index)* * Return: An encoded page directory entry. No errors. */ -u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level cache) +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 pat_index) {u64 pde;struct xe_vm *vm = bo->vm;struct xe_device *xe = vm->xe; - pde =xe_bo_addr (bo, bo_offset,XE_PAGE_SIZE ); pde |=XE_PAGE_PRESENT |XE_PAGE_RW ;if ((vm->pat_encode).pde_encode) - pde = (vm->pat_encode).pde_encode (xe, pde, cache); + pde = (vm->pat_encode).pde_encode (xe, pde, pat_index);return pde; } -static u64 __pte_encode (u64 pte,enum xe_cache_level cache, +static u64 __pte_encode (u64 pte,u32 pat_index,struct xe_vma *vma,u32 pt_level) {struct xe_vm *vm =xe_vma_vm (vma);@@ -89,7 +88,7 @@ static u64 __pte_encode(u64 pte, enum xe_cache_level cache, pte |=XE_PTE_NULL ;if ((vm->pat_encode).pte_encode) - pte = (vm->pat_encode).pte_encode (xe, pte, cache); + pte = (vm->pat_encode).pte_encode (xe, pte, pat_index);if (pt_level ==1 ) pte |=XE_PDE_PS_2M ;@@ -112,7 +111,7 @@ static u64 __pte_encode(u64 pte, enum xe_cache_level cache,* * Return: An encoded page-table entry. No errors. */ -u64 xe_pte_encode (struct xe_bo *bo,u64 offset,enum xe_cache_level cache, +u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 pat_index,u32 pt_level) {u64 pte;@@ -121,7 +120,7 @@ u64 xe_pte_encode(struct xe_bo *bo, u64 offset, enum xe_cache_level cache,if (xe_bo_is_vram (bo) ||xe_bo_is_stolen_devmem (bo)) pte |=XE_PPGTT_PTE_DM ; -return __pte_encode (pte, cache,NULL , pt_level); +return __pte_encode (pte, pat_index,NULL , pt_level); } static u64 __xe_pt_empty_pte(struct xe_tile *tile, struct xe_vm *vm,@@ -134,12 +133,12 @@ static u64 __xe_pt_empty_pte(struct xe_tile *tile, struct xe_vm *vm,if (level ==0 ) {u64 empty =xe_pte_encode (vm->scratch_bo[id],0 , -XE_CACHE_WB ,0 ); +xe_pat_get_index (vm->xe,XE_CACHE_WB ),0 );return empty; }else {return xe_pde_encode (vm->scratch_pt[id][level -1 ]->bo,0 , -XE_CACHE_WB ); +xe_pat_get_index (vm->xe,XE_CACHE_WB )); } }@@ -368,8 +367,6 @@ struct xe_pt_stage_bind_walk {struct xe_vm *vm;/** @tile: The tile we're building for. */ struct xe_tile *tile; -/** @cache: Desired cache level for the ptes */ -enum xe_cache_level cache;/** @default_pte: PTE flag only template. No address is associated */ u64 default_pte;/** @dma_offset: DMA offset to add to the PTE. */ @@ -604,7 +601,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset, pte =__pte_encode (is_null ?0 :xe_res_dma (curs) + xe_walk->dma_offset, - xe_walk->cache, xe_walk->vma, level); + xe_walk->vma->pat_index, xe_walk->vma, level); pte |= xe_walk->default_pte;/* @@ -669,7 +666,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset, xe_child->is_compact = true; } - pte =xe_pde_encode (xe_child->bo,0 , xe_walk->cache) | flags; + pte =xe_pde_encode (xe_child->bo,0 , xe_walk->vma->pat_index) | flags; ret =xe_pt_insert_entry (xe_walk, xe_parent, offset, xe_child, pte); }@@ -730,13 +727,8 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma,if (vma && vma->gpuva.flags &XE_VMA_ATOMIC_PTE_BIT ) xe_walk.default_pte |=XE_USM_PPGTT_PTE_AE ; xe_walk.dma_offset =vram_region_gpu_offset (bo->ttm.resource); - xe_walk.cache =XE_CACHE_WB ; - }else { -if (!xe_vma_has_no_bo (vma) && bo->flags &XE_BO_SCANOUT_BIT ) - xe_walk.cache =XE_CACHE_WT ; -else - xe_walk.cache =XE_CACHE_WB ; } +if (!xe_vma_has_no_bo (vma) &&xe_bo_is_stolen (bo)) xe_walk.dma_offset =xe_ttm_stolen_gpu_offset (xe_bo_device (bo));drivers/gpu/drm/xe/xe_pt.h +2
-4@@ -45,10 +45,8 @@ __xe_pt_unbind_vma(struct xe_tile *tile, struct xe_vma *vma, struct xe_exec_queubool xe_pt_zap_ptes (struct xe_tile *tile,struct xe_vma *vma); -u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level level); +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 pat_index); -u64 xe_pte_encode (struct xe_bo *bo,u64 offset,enum xe_cache_level cache, -u32 pt_level); +u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 pat_index,u32 pt_level);#endif drivers/gpu/drm/xe/xe_vm.c +58
-15@@ -6,6 +6,7 @@#include "xe_vm.h" #include <linux/dma-fence-array.h> +#include <linux/nospec.h> #include <drm/drm_print.h> #include <drm/ttm/ttm_execbuf_util.h> @@ -874,7 +875,8 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, u64 start, u64 end, bool read_only, bool is_null, - u8 tile_mask) + u8 tile_mask, + u32 pat_index) { struct xe_vma *vma;struct xe_tile *tile;@@ -913,6 +915,8 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, vma->tile_mask |=0x1 << id; } + vma->pat_index = pat_index; +if (vm->xe->info.platform ==XE_PVC ) vma->gpuva.flags |=XE_VMA_ATOMIC_PTE_BIT ;@@ -1194,10 +1198,8 @@ static void xe_vma_op_work_func(struct work_struct *w);static void vm_destroy_work_func (struct work_struct *w);static u64 xelp_ppgtt_pde_encode_pat (struct xe_device *xe,u64 pde_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pde_pat &= ~(XELP_PDE_PAT_MASK );if (pat_index &BIT (0 ))@@ -1213,10 +1215,8 @@ static u64 xelp_ppgtt_pde_encode_pat(struct xe_device *xe, u64 pde_pat, }static u64 xelp_ppgtt_pte_encode_pat (struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELP_PTE_PAT_MASK );if (pat_index &BIT (0 ))@@ -1622,7 +1622,7 @@ struct xe_vm *xe_vm_lookup(struct xe_file *xef, u32 id)u64 xe_vm_pdp4_descriptor (struct xe_vm *vm,struct xe_tile *tile) {return xe_pde_encode (vm->pt_root[tile->id]->bo,0 , -XE_CACHE_WB ); +xe_pat_get_index (vm->xe,XE_CACHE_WB )); } static struct dma_fence *@@ -2311,7 +2311,7 @@ static void print_op(struct xe_device *xe, struct drm_gpuva_op *op)static struct drm_gpuva_ops *vm_bind_ioctl_ops_create (struct xe_vm *vm,struct xe_bo *bo,u64 bo_offset_or_userptr,u64 addr,u64 range, -u32 operation,u8 tile_mask,u32 region) +u32 operation,u8 tile_mask,u32 region,u32 pat_index) {struct drm_gem_object *obj = bo ? &bo->ttm.base :NULL ;struct ww_acquire_ctx ww;@@ -2339,6 +2339,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo,struct xe_vma_op *op =gpuva_op_to_vma_op (__op); op->tile_mask = tile_mask; + op->pat_index = pat_index; op->map.immediate = operation &XE_VM_BIND_FLAG_IMMEDIATE ; op->map.read_only =@@ -2366,6 +2367,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo,struct xe_vma_op *op =gpuva_op_to_vma_op (__op); op->tile_mask = tile_mask; + op->pat_index = pat_index; op->prefetch.region = region; }break ;@@ -2408,7 +2410,8 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo, }static struct xe_vma *new_vma (struct xe_vm *vm,struct drm_gpuva_op_map *op, -u8 tile_mask,bool read_only,bool is_null) +u8 tile_mask,bool read_only,bool is_null, +u32 pat_index) {struct xe_bo *bo = op->gem.obj ?gem_to_xe_bo (op->gem.obj) :NULL ;struct xe_vma *vma;@@ -2425,7 +2428,7 @@ static struct xe_vma *new_vma(struct xe_vm *vm, struct drm_gpuva_op_map *op, vma =xe_vma_create (vm, bo, op->gem.offset, op->va.addr, op->va.addr + op->va.range -1 , read_only, is_null, - tile_mask); + tile_mask, pat_index);if (bo)xe_bo_unlock (bo, &ww);@@ -2539,7 +2542,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, &op->base.map, op->tile_mask, op->map.read_only, - op->map.is_null); + op->map.is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2567,7 +2570,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, op->base.remap.prev, op->tile_mask, read_only, - is_null); + is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2603,7 +2606,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, op->base.remap.next, op->tile_mask, read_only, - is_null); + is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -3158,8 +3161,14 @@ static int vm_bind_ioctl_check_args(struct xe_device *xe,u32 obj = (*bind_ops)[i].obj;u64 obj_offset = (*bind_ops)[i].obj_offset;u32 region = (*bind_ops)[i].region; +u32 pat_index = (*bind_ops)[i].pat_index;bool is_null = op &XE_VM_BIND_FLAG_NULL ; +if (XE_IOCTL_DBG (xe, pat_index >= xe->info.pat_table_n_entries)) { + err = -EINVAL ; + goto free_bind_ops; + } +if (i ==0 ) { *async = !!(op &XE_VM_BIND_FLAG_ASYNC ); } else if (XE_IOCTL_DBG(xe, !*async) ||@@ -3346,8 +3355,25 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)struct drm_gem_object *gem_obj;u64 range = bind_ops[i].range;u64 addr = bind_ops[i].addr; +u32 op = bind_ops[i].op;u32 obj = bind_ops[i].obj;u64 obj_offset = bind_ops[i].obj_offset; +u32 pat_index = bind_ops[i].pat_index; +u16 coh_mode; + + pat_index =array_index_nospec (pat_index, + xe->info.pat_table_n_entries); + coh_mode =xe_pat_index_get_coh_mode (xe, pat_index); +if (XE_IOCTL_DBG (xe, !coh_mode)) { + err = -EINVAL ; + goto put_obj; + } + +if (XE_IOCTL_DBG (xe,VM_BIND_OP (op) ==XE_VM_BIND_OP_MAP_USERPTR && + coh_mode ==XE_GEM_COHERENCY_NONE )) { + err = -EINVAL ; + goto put_obj; + }if (!obj)continue ;@@ -3375,6 +3401,22 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) goto put_obj; } } + +if (bos[i]->coh_mode) { +if (XE_IOCTL_DBG (xe, bos[i]->coh_mode != coh_mode)) { + err = -EINVAL ; + goto put_obj; + } + }else if (XE_IOCTL_DBG (xe, coh_mode ==XE_GEM_COHERENCY_NONE )) { +/* +* Imported dma-buf from a different device should +* require 1way or 2way coherency since we don't know +* how it was mapped on CPU. Just assume is it +* potentially cached on CPU side. +*/ + err = -EINVAL ; + goto put_obj; + } }if (args->num_syncs) {@@ -3412,10 +3454,11 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)u64 obj_offset = bind_ops[i].obj_offset;u8 tile_mask = bind_ops[i].tile_mask;u32 region = bind_ops[i].region; +u32 pat_index = bind_ops[i].pat_index; ops[i] =vm_bind_ioctl_ops_create (vm, bos[i], obj_offset, addr, range, op, tile_mask, - region); + region, pat_index);if (IS_ERR (ops[i])) { err =PTR_ERR (ops[i]); ops[i] =NULL ;drivers/gpu/drm/xe/xe_vm_types.h +9
-4@@ -111,6 +111,11 @@ struct xe_vma {*/ u8 tile_present; +/** +* @pat_index: The pat index to use when encoding the PTEs for this vma. +*/ +u32 pat_index; +struct {struct list_head rebind_link; } notifier;@@ -338,10 +343,8 @@ struct xe_vm {bool batch_invalidate_tlb;struct { -u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache); -u64 (*pde_encode)(struct xe_device *xe,u64 pde_pat, -enum xe_cache_level cache); +u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat,u32 pat_index); +u64 (*pde_encode)(struct xe_device *xe,u64 pde_pat,u32 pat_index); } pat_encode; };@@ -417,6 +420,8 @@ struct xe_vma_op {struct async_op_fence *fence;/** @tile_mask: gt mask for this operation */ u8 tile_mask; +/** @pat_index: The pat index to use for this operation. */ +u32 pat_index;/** @flags: operation flags */ enum xe_vma_op_flags flags;include/uapi/drm/xe_drm.h +39
-2@@ -600,8 +600,45 @@ struct drm_xe_vm_bind_op {*/ __u32 obj; -/** @pad: MBZ */ -__u32 pad; +/** +* @pat_index: The platform defined @pat_index to use for this mapping. +* The index basically maps to some predefined memory attributes, +* including things like caching, coherency and likely other stuff in +* the future. The exact meaning of the pat_index is platform specific +* and defined in the Bspec and PRMs. When the KMD sets up the binding +* the index here is encoded into the ppGTT PTE. +* +* For coherency the @pat_index needs to match the +* drm_xe_gem_create.coh_mode, so either XE_GEM_COHERENCY_NONE, +* XE_GEM_COHERENCY_1WAY or XE_GEM_COHERENCY_2WAY. The KMD will extract +* the coherency mode from the @pat_index and reject if there is a +* mismatch (see note below for pre-MTL platforms). +* +* Note: On pre-MTL platforms there is only a caching mode and no +* explicit coherency mode, but on such hardware there is always a +* shared-LLC (or is dgpu) so all GT memory accesses are coherent with +* CPU caches even with the caching mode set as uncached. It's only the +* display engine that is incoherent (on dgpu it must be in VRAM which +* is always mapped as WC on the CPU). However to keep the uapi somewhat +* consistent with newer platforms the KMD groups the different cache +* levels into the following coherency buckets on all pre-MTL platforms: +* +* ppGTT UC -> XE_GEM_COHERENCY_NONE +* ppGTT WC -> XE_GEM_COHERENCY_NONE +* ppGTT WT -> XE_GEM_COHERENCY_NONE +* ppGTT WB -> XE_GEM_COHERENCY_2WAY +* +* In practice UC/WC/WT should only ever used for scanout surfaces on +* such platforms since it is only the display engine that is actually +* incoherent. Everything else should typically use WB given that we +* have a shared-LLC. On MTL+ this completely changes (also potentially +* no shared-LLC) and the HW defines the coherency mode as part of the +* @pat_index. +* +* Note: For userptr and externally imported dma-buf the kernel expects +* either 1WAY or 2WAY for the @pat_index. +*/ +__u32 pat_index;union {/**
Re: [Intel-xe] [RFC 5/5] drm/xe/uapi: support pat_index selection with vm_bind
Quoted 31 lines · Jose
Quoted 30 lines · Matthew Auld
Quoted 28 lines · Jose
Quoted 18 lines · Matthew Auld
Allow userspace to directly control the
pat_indexfor a given vm binding. This should allow directly controlling the coherency, caching and potentially other stuff in the future for the ppGTT binding.The exact meaning behind the
pat_indexis very platform specific (see BSpec or PRMs) but effectively maps to some predefined memory attributes. From the KMD pov we only care about the coherency that is provided by thepat_index, which falls into either NONE, 1WAY or 2WAY. Thevm_bindcoherency mode for the givenpat_indexneeds to match the givencoh_modethat was set at object creation. For platforms that lack the explicit coherency mode, we treat UC/WT/WC as NONE and WB as 2WAY.For userptr mappings we lack a corresponding gem object, so the expected coherency mode is instead implicit and must fall into either 1WAY or 2WAY. Trying to use NONE will be rejected by the kernel. For imported dma-buf (from a different device) the coherency mode is also implicit and must also be either 1WAY or 2WAY.
Quoted 7 lines
As part of adding
pat_indexsupport withvm_bindwe also need stop usingxe_cache_leveland instead use thepat_indexin various places. We still make use ofxe_cache_level, but only as a convenience for kernel internal objectsi (internally it maps to some reasonablepat_index). For now this is just a 1:1 conversion of the existing code, however for platforms like MTL+ we might need to give more control throughbo_createor stop using WB on the CPU side if we need CPU access.
xe_ttm_tt_create()caching should not also take into consideration the PAT index?
Quoted 8 lines
Quoted 5 lines
Which PAT index? At least for userspace objects we might not know the
pat_indexat this stage, also thepat_indexis more per vma, while the CPU caching mode inxe_ttm_tt_create()is per object. At gem create we only need to consider the overallcoh_modeof everypat_indexthat will map the object to determine if the CPU caching mode is "valid".Yeah it is too late.
I should not considersmem_cachingindrm_xe_gem_create? It that what will be used as caching mode for mmap?
You will need pick something for smem_caching, unless it's vram-only in which case just leave as zero. And yes, the mmap caching mode will be whatever you picked for smem_caching.
Quoted 612 lines
Quoted 611 lines
Quoted 611 lines
Quoted 610 lines
Bspec: 45101, 44235 #xe
Bspec: 70552, 71582, 59400 #xe2Signed-off-by Matthew Auld <[email protected]>Cc Pallavi Mishra <[email protected]>Cc Thomas Hellström <[email protected]>Cc Joonas Lahtinen <[email protected]>Cc Lucas De Marchi <[email protected]>Cc Matt Roper <[email protected]>Cc José Roberto de Souza <[email protected]>Cc Filip Hazubski <[email protected]>Cc Carl Zhang <[email protected]>Cc Effie Yu <[email protected]>--- drivers/gpu/drm/i915/gt/intel_gtt.h | 2 +- drivers/gpu/drm/xe/tests/xe_migrate.c | 2 +- drivers/gpu/drm/xe/xe_ggtt.c | 7 ++- drivers/gpu/drm/xe/xe_ggtt_types.h | 2 +- drivers/gpu/drm/xe/xe_migrate.c | 14 ++--- drivers/gpu/drm/xe/xe_pt.c | 32 +++++------- drivers/gpu/drm/xe/xe_pt.h | 6 +-- drivers/gpu/drm/xe/xe_vm.c | 73 +++++++++++++++++++++------ drivers/gpu/drm/xe/xe_vm_types.h | 13 +++-- include/uapi/drm/xe_drm.h | 41 ++++++++++++++- 10 files changed, 134 insertions(+), 58 deletions(-)drivers/gpu/drm/i915/gt/intel_gtt.h +1
-1@@ -302,7 +302,7 @@ struct i915_address_space { (*alloc_scratch_dma)(struct i915_address_space *vm, int sz);u64 (*pte_encode)(dma_addr_t addr, -unsigned int pat_index, +u32 pat_index,u32 flags);/* Create a valid PTE */ #define PTE_READ_ONLY BIT(0)#define PTE_LM BIT(1)drivers/gpu/drm/xe/tests/xe_migrate.c +1
-1@@ -301,7 +301,7 @@ static void xe_migrate_sanity_test(struct xe_migrate *m, struct kunit *test)/* First part of the test, are we updating our pagetable bo with a new entry? */ xe_map_wr (xe, &bo->vmap,XE_PAGE_SIZE * (NUM_KERNEL_PDE -1 ), u64,0xdeaddeadbeefbeef ); - expected =xe_pte_encode (pt,0 ,XE_CACHE_WB ,0 ); + expected =xe_pte_encode (pt,0 ,xe_pat_get_index (xe,XE_CACHE_WB ),0 );if (m->q->vm->flags &XE_VM_FLAG_64K ) expected |=XE_PTE_PS64 ;if (xe_bo_is_vram (pt))drivers/gpu/drm/xe/xe_ggtt.c +3
-4@@ -41,7 +41,8 @@ u64 xe_ggtt_pte_encode(struct xe_bo *bo, u64 bo_offset) pte |=XE_GGTT_PTE_DM ;if ((ggtt->pat_encode).pte_encode) - pte = (ggtt->pat_encode).pte_encode (xe, pte,XE_CACHE_WB_1_WAY ); + pte = (ggtt->pat_encode).pte_encode (xe, pte, +xe_pat_get_index (xe,XE_CACHE_WB_1_WAY ));return pte; }@@ -102,10 +103,8 @@ static void primelockdep(struct xe_ggtt *ggtt) }static u64 xelpg_ggtt_pte_encode_pat (struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELPG_GGTT_PTE_PAT_MASK );if (pat_index &BIT (0 ))drivers/gpu/drm/xe/xe_ggtt_types.h +1
-1@@ -31,7 +31,7 @@ struct xe_ggtt {struct {u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache); +u32 pat_index); } pat_encode; };drivers/gpu/drm/xe/xe_migrate.c +8
-6@@ -24,6 +24,7 @@#include "xe_lrc.h" #include "xe_map.h" #include "xe_mocs.h" +#include "xe_pat.h" #include "xe_pt.h" #include "xe_res_cursor.h" #include "xe_sched_job.h" @@ -162,6 +163,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,u32 num_entries =NUM_PT_SLOTS , num_level = vm->pt_root[id]->level;u32 map_ofs, level, i;struct xe_bo *bo, *batch = tile->mem.kernel_bb_pool->bo; +u32 pat_index =xe_pat_get_index (xe,XE_CACHE_WB );u64 entry;int ret;@@ -189,14 +191,14 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,return ret; } - entry =xe_pde_encode (bo, bo->size -XE_PAGE_SIZE ,XE_CACHE_WB ); + entry =xe_pde_encode (bo, bo->size -XE_PAGE_SIZE , pat_index);xe_pt_write (xe, &vm->pt_root[id]->bo->vmap,0 , entry); map_ofs = (num_entries - num_level) *XE_PAGE_SIZE ;/* Map the entire BO in our level 0 pt */ for (i =0 , level =0 ; i < num_entries; level++) { - entry =xe_pte_encode (bo, i *XE_PAGE_SIZE ,XE_CACHE_WB ,0 ); + entry =xe_pte_encode (bo, i *XE_PAGE_SIZE , pat_index,0 );xe_map_wr (xe, &bo->vmap, map_ofs + level *8 , u64, entry);@@ -214,7 +216,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,for (i =0 ; i < batch->size; i += vm->flags &XE_VM_FLAG_64K ?XE_64K_PAGE_SIZE :XE_PAGE_SIZE ) { - entry =xe_pte_encode (batch, i,XE_CACHE_WB ,0 ); + entry =xe_pte_encode (batch, i, pat_index,0 );xe_map_wr (xe, &bo->vmap, map_ofs + level *8 , u64, entry);@@ -239,7 +241,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m, flags =XE_PDE_64K ; entry =xe_pde_encode (bo, map_ofs + (level -1 ) * -XE_PAGE_SIZE ,XE_CACHE_WB ); +XE_PAGE_SIZE , pat_index);xe_map_wr (xe, &bo->vmap, map_ofs +XE_PAGE_SIZE * level, u64, entry | flags); }@@ -247,7 +249,7 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,/* Write PDE's that point to our BO. */ for (i =0 ; i < num_entries - num_level; i++) { entry =xe_pde_encode (bo, i *XE_PAGE_SIZE , -XE_CACHE_WB ); + pat_index);xe_map_wr (xe, &bo->vmap, map_ofs +XE_PAGE_SIZE + (i +1 ) *8 , u64, entry);@@ -1256,7 +1258,7 @@ xe_migrate_update_pgtables(struct xe_migrate *m,XE_WARN_ON (pt_bo->size !=SZ_4K ); - addr =xe_pte_encode (pt_bo,0 ,XE_CACHE_WB ,0 ); + addr =xe_pte_encode (pt_bo,0 ,xe_pat_get_index (xe,XE_CACHE_WB ),0 ); bb->cs[bb->len++] =lower_32_bits (addr); bb->cs[bb->len++] =upper_32_bits (addr); }drivers/gpu/drm/xe/xe_pt.c +12
-20@@ -10,6 +10,7 @@#include "xe_gt.h" #include "xe_gt_tlb_invalidation.h" #include "xe_migrate.h" +#include "xe_pat.h" #include "xe_pt_types.h" #include "xe_pt_walk.h" #include "xe_res_cursor.h" @@ -57,24 +58,22 @@ static struct xe_pt *xe_pt_entry(struct xe_pt_dir *pt_dir, unsigned int index)* * Return: An encoded page directory entry. No errors. */ -u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level cache) +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 pat_index) {u64 pde;struct xe_vm *vm = bo->vm;struct xe_device *xe = vm->xe; - pde =xe_bo_addr (bo, bo_offset,XE_PAGE_SIZE ); pde |=XE_PAGE_PRESENT |XE_PAGE_RW ;if ((vm->pat_encode).pde_encode) - pde = (vm->pat_encode).pde_encode (xe, pde, cache); + pde = (vm->pat_encode).pde_encode (xe, pde, pat_index);return pde; } -static u64 __pte_encode (u64 pte,enum xe_cache_level cache, +static u64 __pte_encode (u64 pte,u32 pat_index,struct xe_vma *vma,u32 pt_level) {struct xe_vm *vm =xe_vma_vm (vma);@@ -89,7 +88,7 @@ static u64 __pte_encode(u64 pte, enum xe_cache_level cache, pte |=XE_PTE_NULL ;if ((vm->pat_encode).pte_encode) - pte = (vm->pat_encode).pte_encode (xe, pte, cache); + pte = (vm->pat_encode).pte_encode (xe, pte, pat_index);if (pt_level ==1 ) pte |=XE_PDE_PS_2M ;@@ -112,7 +111,7 @@ static u64 __pte_encode(u64 pte, enum xe_cache_level cache,* * Return: An encoded page-table entry. No errors. */ -u64 xe_pte_encode (struct xe_bo *bo,u64 offset,enum xe_cache_level cache, +u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 pat_index,u32 pt_level) {u64 pte;@@ -121,7 +120,7 @@ u64 xe_pte_encode(struct xe_bo *bo, u64 offset, enum xe_cache_level cache,if (xe_bo_is_vram (bo) ||xe_bo_is_stolen_devmem (bo)) pte |=XE_PPGTT_PTE_DM ; -return __pte_encode (pte, cache,NULL , pt_level); +return __pte_encode (pte, pat_index,NULL , pt_level); } static u64 __xe_pt_empty_pte(struct xe_tile *tile, struct xe_vm *vm,@@ -134,12 +133,12 @@ static u64 __xe_pt_empty_pte(struct xe_tile *tile, struct xe_vm *vm,if (level ==0 ) {u64 empty =xe_pte_encode (vm->scratch_bo[id],0 , -XE_CACHE_WB ,0 ); +xe_pat_get_index (vm->xe,XE_CACHE_WB ),0 );return empty; }else {return xe_pde_encode (vm->scratch_pt[id][level -1 ]->bo,0 , -XE_CACHE_WB ); +xe_pat_get_index (vm->xe,XE_CACHE_WB )); } }@@ -368,8 +367,6 @@ struct xe_pt_stage_bind_walk {struct xe_vm *vm;/** @tile: The tile we're building for. */ struct xe_tile *tile; -/** @cache: Desired cache level for the ptes */ -enum xe_cache_level cache;/** @default_pte: PTE flag only template. No address is associated */ u64 default_pte;/** @dma_offset: DMA offset to add to the PTE. */ @@ -604,7 +601,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset, pte =__pte_encode (is_null ?0 :xe_res_dma (curs) + xe_walk->dma_offset, - xe_walk->cache, xe_walk->vma, level); + xe_walk->vma->pat_index, xe_walk->vma, level); pte |= xe_walk->default_pte;/* @@ -669,7 +666,7 @@ xe_pt_stage_bind_entry(struct xe_ptw *parent, pgoff_t offset, xe_child->is_compact = true; } - pte =xe_pde_encode (xe_child->bo,0 , xe_walk->cache) | flags; + pte =xe_pde_encode (xe_child->bo,0 , xe_walk->vma->pat_index) | flags; ret =xe_pt_insert_entry (xe_walk, xe_parent, offset, xe_child, pte); }@@ -730,13 +727,8 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma,if (vma && vma->gpuva.flags &XE_VMA_ATOMIC_PTE_BIT ) xe_walk.default_pte |=XE_USM_PPGTT_PTE_AE ; xe_walk.dma_offset =vram_region_gpu_offset (bo->ttm.resource); - xe_walk.cache =XE_CACHE_WB ; - }else { -if (!xe_vma_has_no_bo (vma) && bo->flags &XE_BO_SCANOUT_BIT ) - xe_walk.cache =XE_CACHE_WT ; -else - xe_walk.cache =XE_CACHE_WB ; } +if (!xe_vma_has_no_bo (vma) &&xe_bo_is_stolen (bo)) xe_walk.dma_offset =xe_ttm_stolen_gpu_offset (xe_bo_device (bo));drivers/gpu/drm/xe/xe_pt.h +2
-4@@ -45,10 +45,8 @@ __xe_pt_unbind_vma(struct xe_tile *tile, struct xe_vma *vma, struct xe_exec_queubool xe_pt_zap_ptes (struct xe_tile *tile,struct xe_vma *vma); -u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset, -const enum xe_cache_level level); +u64 xe_pde_encode (struct xe_bo *bo,u64 bo_offset,u32 pat_index); -u64 xe_pte_encode (struct xe_bo *bo,u64 offset,enum xe_cache_level cache, -u32 pt_level); +u64 xe_pte_encode (struct xe_bo *bo,u64 offset,u32 pat_index,u32 pt_level);#endif drivers/gpu/drm/xe/xe_vm.c +58
-15@@ -6,6 +6,7 @@#include "xe_vm.h" #include <linux/dma-fence-array.h> +#include <linux/nospec.h> #include <drm/drm_print.h> #include <drm/ttm/ttm_execbuf_util.h> @@ -874,7 +875,8 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, u64 start, u64 end, bool read_only, bool is_null, - u8 tile_mask) + u8 tile_mask, + u32 pat_index) { struct xe_vma *vma;struct xe_tile *tile;@@ -913,6 +915,8 @@ static struct xe_vma *xe_vma_create(struct xe_vm *vm, vma->tile_mask |=0x1 << id; } + vma->pat_index = pat_index; +if (vm->xe->info.platform ==XE_PVC ) vma->gpuva.flags |=XE_VMA_ATOMIC_PTE_BIT ;@@ -1194,10 +1198,8 @@ static void xe_vma_op_work_func(struct work_struct *w);static void vm_destroy_work_func (struct work_struct *w);static u64 xelp_ppgtt_pde_encode_pat (struct xe_device *xe,u64 pde_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pde_pat &= ~(XELP_PDE_PAT_MASK );if (pat_index &BIT (0 ))@@ -1213,10 +1215,8 @@ static u64 xelp_ppgtt_pde_encode_pat(struct xe_device *xe, u64 pde_pat, }static u64 xelp_ppgtt_pte_encode_pat (struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache) +u32 pat_index) { -u32 pat_index =xe_pat_get_index (xe, cache); - pte_pat &= ~(XELP_PTE_PAT_MASK );if (pat_index &BIT (0 ))@@ -1622,7 +1622,7 @@ struct xe_vm *xe_vm_lookup(struct xe_file *xef, u32 id)u64 xe_vm_pdp4_descriptor (struct xe_vm *vm,struct xe_tile *tile) {return xe_pde_encode (vm->pt_root[tile->id]->bo,0 , -XE_CACHE_WB ); +xe_pat_get_index (vm->xe,XE_CACHE_WB )); } static struct dma_fence *@@ -2311,7 +2311,7 @@ static void print_op(struct xe_device *xe, struct drm_gpuva_op *op)static struct drm_gpuva_ops *vm_bind_ioctl_ops_create (struct xe_vm *vm,struct xe_bo *bo,u64 bo_offset_or_userptr,u64 addr,u64 range, -u32 operation,u8 tile_mask,u32 region) +u32 operation,u8 tile_mask,u32 region,u32 pat_index) {struct drm_gem_object *obj = bo ? &bo->ttm.base :NULL ;struct ww_acquire_ctx ww;@@ -2339,6 +2339,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo,struct xe_vma_op *op =gpuva_op_to_vma_op (__op); op->tile_mask = tile_mask; + op->pat_index = pat_index; op->map.immediate = operation &XE_VM_BIND_FLAG_IMMEDIATE ; op->map.read_only =@@ -2366,6 +2367,7 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo,struct xe_vma_op *op =gpuva_op_to_vma_op (__op); op->tile_mask = tile_mask; + op->pat_index = pat_index; op->prefetch.region = region; }break ;@@ -2408,7 +2410,8 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_bo *bo, }static struct xe_vma *new_vma (struct xe_vm *vm,struct drm_gpuva_op_map *op, -u8 tile_mask,bool read_only,bool is_null) +u8 tile_mask,bool read_only,bool is_null, +u32 pat_index) {struct xe_bo *bo = op->gem.obj ?gem_to_xe_bo (op->gem.obj) :NULL ;struct xe_vma *vma;@@ -2425,7 +2428,7 @@ static struct xe_vma *new_vma(struct xe_vm *vm, struct drm_gpuva_op_map *op, vma =xe_vma_create (vm, bo, op->gem.offset, op->va.addr, op->va.addr + op->va.range -1 , read_only, is_null, - tile_mask); + tile_mask, pat_index);if (bo)xe_bo_unlock (bo, &ww);@@ -2539,7 +2542,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, &op->base.map, op->tile_mask, op->map.read_only, - op->map.is_null); + op->map.is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2567,7 +2570,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, op->base.remap.prev, op->tile_mask, read_only, - is_null); + is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -2603,7 +2606,7 @@ static int vm_bind_ioctl_ops_parse(struct xe_vm *vm, struct xe_exec_queue *q, vma =new_vma (vm, op->base.remap.next, op->tile_mask, read_only, - is_null); + is_null, op->pat_index);if (IS_ERR (vma)) { err =PTR_ERR (vma); goto free_fence;@@ -3158,8 +3161,14 @@ static int vm_bind_ioctl_check_args(struct xe_device *xe,u32 obj = (*bind_ops)[i].obj;u64 obj_offset = (*bind_ops)[i].obj_offset;u32 region = (*bind_ops)[i].region; +u32 pat_index = (*bind_ops)[i].pat_index;bool is_null = op &XE_VM_BIND_FLAG_NULL ; +if (XE_IOCTL_DBG (xe, pat_index >= xe->info.pat_table_n_entries)) { + err = -EINVAL ; + goto free_bind_ops; + } +if (i ==0 ) { *async = !!(op &XE_VM_BIND_FLAG_ASYNC ); } else if (XE_IOCTL_DBG(xe, !*async) ||@@ -3346,8 +3355,25 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)struct drm_gem_object *gem_obj;u64 range = bind_ops[i].range;u64 addr = bind_ops[i].addr; +u32 op = bind_ops[i].op;u32 obj = bind_ops[i].obj;u64 obj_offset = bind_ops[i].obj_offset; +u32 pat_index = bind_ops[i].pat_index; +u16 coh_mode; + + pat_index =array_index_nospec (pat_index, + xe->info.pat_table_n_entries); + coh_mode =xe_pat_index_get_coh_mode (xe, pat_index); +if (XE_IOCTL_DBG (xe, !coh_mode)) { + err = -EINVAL ; + goto put_obj; + } + +if (XE_IOCTL_DBG (xe,VM_BIND_OP (op) ==XE_VM_BIND_OP_MAP_USERPTR && + coh_mode ==XE_GEM_COHERENCY_NONE )) { + err = -EINVAL ; + goto put_obj; + }if (!obj)continue ;@@ -3375,6 +3401,22 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) goto put_obj; } } + +if (bos[i]->coh_mode) { +if (XE_IOCTL_DBG (xe, bos[i]->coh_mode != coh_mode)) { + err = -EINVAL ; + goto put_obj; + } + }else if (XE_IOCTL_DBG (xe, coh_mode ==XE_GEM_COHERENCY_NONE )) { +/* +* Imported dma-buf from a different device should +* require 1way or 2way coherency since we don't know +* how it was mapped on CPU. Just assume is it +* potentially cached on CPU side. +*/ + err = -EINVAL ; + goto put_obj; + } }if (args->num_syncs) {@@ -3412,10 +3454,11 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file)u64 obj_offset = bind_ops[i].obj_offset;u8 tile_mask = bind_ops[i].tile_mask;u32 region = bind_ops[i].region; +u32 pat_index = bind_ops[i].pat_index; ops[i] =vm_bind_ioctl_ops_create (vm, bos[i], obj_offset, addr, range, op, tile_mask, - region); + region, pat_index);if (IS_ERR (ops[i])) { err =PTR_ERR (ops[i]); ops[i] =NULL ;drivers/gpu/drm/xe/xe_vm_types.h +9
-4@@ -111,6 +111,11 @@ struct xe_vma {*/ u8 tile_present; +/** +* @pat_index: The pat index to use when encoding the PTEs for this vma. +*/ +u32 pat_index; +struct {struct list_head rebind_link; } notifier;@@ -338,10 +343,8 @@ struct xe_vm {bool batch_invalidate_tlb;struct { -u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat, -enum xe_cache_level cache); -u64 (*pde_encode)(struct xe_device *xe,u64 pde_pat, -enum xe_cache_level cache); +u64 (*pte_encode)(struct xe_device *xe,u64 pte_pat,u32 pat_index); +u64 (*pde_encode)(struct xe_device *xe,u64 pde_pat,u32 pat_index); } pat_encode; };@@ -417,6 +420,8 @@ struct xe_vma_op {struct async_op_fence *fence;/** @tile_mask: gt mask for this operation */ u8 tile_mask; +/** @pat_index: The pat index to use for this operation. */ +u32 pat_index;/** @flags: operation flags */ enum xe_vma_op_flags flags;include/uapi/drm/xe_drm.h +39
-2@@ -600,8 +600,45 @@ struct drm_xe_vm_bind_op {*/ __u32 obj; -/** @pad: MBZ */ -__u32 pad; +/** +* @pat_index: The platform defined @pat_index to use for this mapping. +* The index basically maps to some predefined memory attributes, +* including things like caching, coherency and likely other stuff in +* the future. The exact meaning of the pat_index is platform specific +* and defined in the Bspec and PRMs. When the KMD sets up the binding +* the index here is encoded into the ppGTT PTE. +* +* For coherency the @pat_index needs to match the +* drm_xe_gem_create.coh_mode, so either XE_GEM_COHERENCY_NONE, +* XE_GEM_COHERENCY_1WAY or XE_GEM_COHERENCY_2WAY. The KMD will extract +* the coherency mode from the @pat_index and reject if there is a +* mismatch (see note below for pre-MTL platforms). +* +* Note: On pre-MTL platforms there is only a caching mode and no +* explicit coherency mode, but on such hardware there is always a +* shared-LLC (or is dgpu) so all GT memory accesses are coherent with +* CPU caches even with the caching mode set as uncached. It's only the +* display engine that is incoherent (on dgpu it must be in VRAM which +* is always mapped as WC on the CPU). However to keep the uapi somewhat +* consistent with newer platforms the KMD groups the different cache +* levels into the following coherency buckets on all pre-MTL platforms: +* +* ppGTT UC -> XE_GEM_COHERENCY_NONE +* ppGTT WC -> XE_GEM_COHERENCY_NONE +* ppGTT WT -> XE_GEM_COHERENCY_NONE +* ppGTT WB -> XE_GEM_COHERENCY_2WAY +* +* In practice UC/WC/WT should only ever used for scanout surfaces on +* such platforms since it is only the display engine that is actually +* incoherent. Everything else should typically use WB given that we +* have a shared-LLC. On MTL+ this completely changes (also potentially +* no shared-LLC) and the HW defines the coherency mode as part of the +* @pat_index. +* +* Note: For userptr and externally imported dma-buf the kernel expects +* either 1WAY or 2WAY for the @pat_index. +*/ +__u32 pat_index;union {/**
Patchwork
CI[Intel-xe] ✗ CI.Patch_applied: failure for PAT and cache coherency support
In reply to Matthew Auld== Series Details ==
== Series Details ==
Series: PAT and cache coherency support
URL : https://patchwork.freedesktop.org/series/123027/
State : failure
== Summary ==
=== Applying kernel patches on branch 'drm-xe-next' with base: === Base commit: d8c08057a drm/xe: Add patch version on guc firmware init === git am output follows ===
error: patch failed: drivers/gpu/drm/xe/xe_pat.c:135
error: drivers/gpu/drm/xe/xe_pat.c: patch does not apply
error: patch failed: drivers/gpu/drm/xe/xe_pat.h:26
error: drivers/gpu/drm/xe/xe_pat.h: patch does not apply
hint: Use 'git am --show-current-patch' to see the failed patch
Applying: drm/xe/uapi: Add support for cache and coherency mode
Applying: drm/xe: fix has_llc on rkl
Applying: drm/xe: move pat_table into device info
Patch failed at 0003 drm/xe: move pat_table into device info When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort".
Quoted 3 lines · Matthew Auld
Series still needs bunch of testing, but sending as RFC to get early feedback while the IGT bits are being worked on. Note that the series also directly depends on the WIP series here: https://patchwork.freedesktop.org/series/122708/
Left some comments in the patches but can't start experimenting because https://patchwork.freedesktop.org/series/122708/ breaks driver load. xe_pte_encode() calls __pte_encode() with a NULL vma.
Quoted 17 lines
Goal here is to allow userspace to directly control the
pat_indexwhen mapping memory via the ppGTT, in addtion to the CPU caching mode for system memory. This is very much needed on newer igpu platforms which allow incoherent GT access, where the choice over the cache level and expected coherency is best left to userspace depending on their usecase. In the future there may also be other stuff encoded in thepat_index, so giving userspace direct control will also be needed there.To support this we added new
gem_createuAPI for selecting the CPU cache mode to use for system memory, including the expected GPU coherency mode. There are various restrictions here for the selected coherency mode and compatible CPU cache modes. With that in place the actualpat_indexcan now be provided as part ofvm_bind. The only restriction is that the coherency mode of thepat_indexmust match thegem_createcoherency mode. There are also some special cases like with userptr and dma-buf.
Quoted 7 lines · Jose
Quoted 3 lines · Matthew Auld
Series still needs bunch of testing, but sending as RFC to get early feedback while the IGT bits are being worked on. Note that the series also directly depends on the WIP series here: https://patchwork.freedesktop.org/series/122708/
Left some comments in the patches but can't start experimenting because https://patchwork.freedesktop.org/series/122708/ breaks driver load.
xe_pte_encode()calls__pte_encode()with a NULL vma.
Thanks for the comments.
Yeah, mostly just build tested so far. Once it makes it out of RFC and the uAPI is mostly stable we will share a branch.
Quoted 19 lines
Quoted 18 lines
Goal here is to allow userspace to directly control the
pat_indexwhen mapping memory via the ppGTT, in addtion to the CPU caching mode for system memory. This is very much needed on newer igpu platforms which allow incoherent GT access, where the choice over the cache level and expected coherency is best left to userspace depending on their usecase. In the future there may also be other stuff encoded in thepat_index, so giving userspace direct control will also be needed there.To support this we added new
gem_createuAPI for selecting the CPU cache mode to use for system memory, including the expected GPU coherency mode. There are various restrictions here for the selected coherency mode and compatible CPU cache modes. With that in place the actualpat_indexcan now be provided as part ofvm_bind. The only restriction is that the coherency mode of thepat_indexmust match thegem_createcoherency mode. There are also some special cases like with userptr and dma-buf.