Try to handle TLB invalidations from CT fast-path
Subsystems (2)
v5 cover Try to handle TLB invalidations from CT fast-path
In various test cases and workloads that put the system under a heavy load, we can sometimes see errors with missed TLB invalidations. If we handle the TLB completions directly from IRQ we can see a big improvement.
v4:
- Various tweaks. Big thing is to make sure we grab the fast_lock for the g2h reservation when doing the CT send, which closes one big race with the CT fast-path.
v5:
- Various improvements. Fix the race with adding a fence after its seqno has already been signalled. (Matt Brost)
Signature · 1 line
2.41.0
PATCH v5 1/10 drm/xe/tlb: drop unnecessary smp_wmb()
wake_up_all() and wait_event_timeout() already have the correct barriers as per https://www.kernel.org/doc/Documentation/memory-barriers.txt.
This should ensure that the seqno_recv write can't be re-ordered wrt to the actual wake_up_all() i.e we get woken up but there is no write. The reader side with wait_event_timeout() also has the correct barriers. With that drop the hand rolled smp_wmb(), which is anyway missing some kind of matching barrier on the reader side.
--- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c +4 -1
@@ -333,8 +333,11 @@ int xe_guc_tlb_invalidation_done_handler(struct xe_guc *guc, u32 *msg, u32 len) expected_seqno, msg[0]); } +/* +* wake_up_all() and wait_event_timeout() already have the correct +* barriers. +*/ gt->tlb_invalidation.seqno_recv = msg[0 ]; -smp_wmb ();wake_up_all (&guc->ct.wq); fence = list_first_entry_or_null(>->tlb_invalidation.pending_fences,
Signature · 1 line
2.41.0
PATCH v5 2/10 drm/xe/tlb: ensure we access seqno_recv once
In reply to Matthew AuldEnsure we load gt->tlb_invalidation.seqno_recv once, and use that for our seqno checking. The gt->tlb_invalidation_seqno_past is a shared global variable and can potentially change at any point here. However the checks here need to operate on a stable version of seqno_recv for this to make any sense.
--- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c +6 -6
@@ -257,15 +257,15 @@ int xe_gt_tlb_invalidation_vma(struct xe_gt *gt,static bool tlb_invalidation_seqno_past (struct xe_gt *gt,int seqno) { -if (seqno - gt->tlb_invalidation.seqno_recv < - -(TLB_INVALIDATION_SEQNO_MAX /2 )) +int seqno_recv =READ_ONCE (gt->tlb_invalidation.seqno_recv); + +if (seqno - seqno_recv < -(TLB_INVALIDATION_SEQNO_MAX /2 ))return false; -if (seqno - gt->tlb_invalidation.seqno_recv > - (TLB_INVALIDATION_SEQNO_MAX /2 )) +if (seqno - seqno_recv > (TLB_INVALIDATION_SEQNO_MAX /2 ))return true; -return gt->tlb_invalidation.seqno_recv >= seqno; +return seqno_recv >= seqno; }/**
@@ -337,7 +337,7 @@ int xe_guc_tlb_invalidation_done_handler(struct xe_guc *guc, u32 *msg, u32 len)* wake_up_all() and wait_event_timeout() already have the correct * barriers. */ - gt->tlb_invalidation.seqno_recv = msg[0 ]; +WRITE_ONCE (gt->tlb_invalidation.seqno_recv, msg[0 ]);wake_up_all (&guc->ct.wq); fence = list_first_entry_or_null(>->tlb_invalidation.pending_fences,
Signature · 1 line
2.41.0
PATCH v5 3/10 drm/xe: hold mem_access.ref for CT fast-path
In reply to Matthew AuldJust checking xe_device_mem_access_ongoing() is not enough, we also need to hold the reference otherwise the ref can transition from 1 -> 0 as we enter g2h_read(), leading to warnings. While we can't do a full rpm sync in the IRQ, we can keep the device awake if the ref is non-zero. Introduce a new helper for this and set it to work in for the CT fast-path.
--- drivers/gpu/drm/xe/xe_device.c | 5 +++++ drivers/gpu/drm/xe/xe_device.h | 1 + drivers/gpu/drm/xe/xe_guc_ct.c | 5 ++++- 3 files changed, 10 insertions(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_device.c +5 -0
@@ -412,6 +412,11 @@ u32 xe_device_ccs_bytes(struct xe_device *xe, u64 size) DIV_ROUND_UP(size, NUM_BYTES_PER_CCS_BYTE) : 0; } +bool xe_device_mem_access_get_if_ongoing (struct xe_device *xe) +{ +return atomic_inc_not_zero (&xe->mem_access.ref); +} +void xe_device_mem_access_get (struct xe_device *xe) {bool resumed =xe_pm_runtime_resume_if_suspended (xe);
drivers/gpu/drm/xe/xe_device.h +1 -0
@@ -138,6 +138,7 @@ static inline struct xe_force_wake * gt_to_fw(struct xe_gt *gt) }void xe_device_mem_access_get (struct xe_device *xe); +bool xe_device_mem_access_get_if_ongoing (struct xe_device *xe);void xe_device_mem_access_put (struct xe_device *xe);static inline bool xe_device_mem_access_ongoing (struct xe_device *xe)
drivers/gpu/drm/xe/xe_guc_ct.c +4 -1
@@ -1044,7 +1044,8 @@ void xe_guc_ct_fast_path(struct xe_guc_ct *ct)struct xe_device *xe =ct_to_xe (ct);int len; -if (!xe_device_in_fault_mode (xe) || !xe_device_mem_access_ongoing (xe)) +if (!xe_device_in_fault_mode (xe) || + !xe_device_mem_access_get_if_ongoing (xe))return ;spin_lock (&ct->fast_lock);
@@ -1054,6 +1055,8 @@ void xe_guc_ct_fast_path(struct xe_guc_ct *ct)g2h_fast_path (ct, ct->fast_msg, len); }while (len >0 );spin_unlock (&ct->fast_lock); + +xe_device_mem_access_put (xe); }/* Returns less than zero on error, 0 on done, 1 on more available */
Signature · 1 line
2.41.0
PATCH v5 4/10 drm/xe/ct: hold fast_lock when reserving space for g2h
In reply to Matthew AuldReserving and checking for space on the g2h side relies on the fast_lock, and not the CT lock since we need to release space from the fast CT path. Make sure we hold it when checking for space and reserving it. The main concern is calling __g2h_release_space() as we are reserving something and since the info.space and info.g2h_outstanding operations are not atomic we can get some nonsense values back.
--- drivers/gpu/drm/xe/xe_guc_ct.c | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-)
drivers/gpu/drm/xe/xe_guc_ct.c +15 -7
@@ -346,7 +346,10 @@ static bool h2g_has_room(struct xe_guc_ct *ct, u32 cmd_len)static bool g2h_has_room (struct xe_guc_ct *ct,u32 g2h_len) { -lockdep_assert_held (&ct->lock); +if (!g2h_len) +return true; + +lockdep_assert_held (&ct->fast_lock);return ct->ctbs.g2h.info.space > g2h_len; }
@@ -367,15 +370,15 @@ static void h2g_reserve_space(struct xe_guc_ct *ct, u32 cmd_len) ct->ctbs.h2g.info.space -= cmd_len; } -static void g2h_reserve_space (struct xe_guc_ct *ct,u32 g2h_len,u32 num_g2h) +static void __g2h_reserve_space (struct xe_guc_ct *ct,u32 g2h_len,u32 num_g2h) {XE_BUG_ON (g2h_len > ct->ctbs.g2h.info.space);if (g2h_len) { -spin_lock_irq (&ct->fast_lock); +lockdep_assert_held (&ct->fast_lock); + ct->ctbs.g2h.info.space -= g2h_len; ct->g2h_outstanding += num_g2h; -spin_unlock_irq (&ct->fast_lock); } }
@@ -505,21 +508,26 @@ static int __guc_ct_send_locked(struct xe_guc_ct *ct, const u32 *action, } } +if (g2h_len) +spin_lock_irq (&ct->fast_lock); retry: ret =has_room (ct, len +GUC_CTB_HDR_LEN , g2h_len);if (unlikely (ret)) - goto out; + goto out_unlock; ret =h2g_write (ct, action, len, g2h_fence ? g2h_fence->seqno :0 , !!g2h_fence);if (unlikely (ret)) {if (ret == -EAGAIN ) goto retry; - goto out; + goto out_unlock; } -g2h_reserve_space (ct, g2h_len, num_g2h); +__g2h_reserve_space (ct, g2h_len, num_g2h);xe_guc_notify (ct_to_guc (ct)); +out_unlock: +if (g2h_len) +spin_unlock_irq (&ct->fast_lock); out:return ret; }
Signature · 1 line
2.41.0
PATCH v5 5/10 drm/xe/tlb: increment next seqno after successful CT send
In reply to Matthew AuldIf we are in the middle of a GT reset or similar the CT might be disabled, such that the CT send fails. However we already incremented gt->tlb_invalidation.seqno which might lead to warnings, since we effectively just skipped a seqno:
0000:00:02.0: drm_WARN_ON(expected_seqno != msg[0])
--- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c +6 -5
@@ -124,10 +124,6 @@ static int send_tlb_invalidation(struct xe_guc *guc,trace_xe_gt_tlb_invalidation_fence_send (fence); } action[1 ] = seqno; - gt->tlb_invalidation.seqno = (gt->tlb_invalidation.seqno +1 ) % -TLB_INVALIDATION_SEQNO_MAX ; -if (!gt->tlb_invalidation.seqno) - gt->tlb_invalidation.seqno =1 ; ret =xe_guc_ct_send_locked (&guc->ct, action, len,G2H_LEN_DW_TLB_INVALIDATE ,1 );if (!ret && fence) {
@@ -137,8 +133,13 @@ static int send_tlb_invalidation(struct xe_guc *guc, >->tlb_invalidation.fence_tdr, TLB_TIMEOUT); } -if (!ret) +if (!ret) { + gt->tlb_invalidation.seqno = (gt->tlb_invalidation.seqno +1 ) % +TLB_INVALIDATION_SEQNO_MAX ; +if (!gt->tlb_invalidation.seqno) + gt->tlb_invalidation.seqno =1 ; ret = seqno; + }if (ret <0 && fence)invalidation_fence_signal (fence);mutex_unlock (&guc->ct.lock);
Signature · 1 line
2.41.0
PATCH v5 7/10 drm/xe/gt: tweak placement for signalling TLB fences after GT reset
In reply to Matthew AuldAssumption here is that submission is disabled along with CT, and full GT reset will also nuke TLBs, so should be safe to signal all in-flight TLB fences, but only after the actual reset so move the placement slightly.
--- drivers/gpu/drm/xe/xe_gt.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_gt.c +2 -1
@@ -519,7 +519,6 @@ static int gt_reset(struct xe_gt *gt)xe_uc_stop_prepare (>->uc);xe_gt_pagefault_reset (gt); -xe_gt_tlb_invalidation_reset (gt); err =xe_uc_stop (>->uc);if (err)
@@ -529,6 +528,8 @@ static int gt_reset(struct xe_gt *gt)if (err) goto err_out; +xe_gt_tlb_invalidation_reset (gt); + err =do_gt_restart (gt);if (err) goto err_out;
Signature · 1 line
2.41.0
PATCH v5 8/10 drm/xe/tlb: also update seqno_recv during reset
In reply to Matthew AuldWe might have various kworkers waiting for TLB flushes to complete which are not tracked with an explicit TLB fence, however at this stage that will never happen since the CT is already disabled, so make sure we signal them here under the assumption that we have completed a full GT reset.
--- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c +17 -2
@@ -86,13 +86,28 @@ invalidation_fence_signal(struct xe_gt_tlb_invalidation_fence *fence)* * Signal any pending invalidation fences, should be called during a GT reset */ -void xe_gt_tlb_invalidation_reset (struct xe_gt *gt) +void xe_gt_tlb_invalidation_reset (struct xe_gt *gt) {struct xe_gt_tlb_invalidation_fence *fence, *next; +struct xe_guc *guc = >->uc.guc; +/* +* CT channel is already disabled at this point. No new TLB requests can +* appear. +*/ + +mutex_lock (>->uc.guc.ct.lock);cancel_delayed_work (>->tlb_invalidation.fence_tdr); +/* +* We might have various kworkers waiting for TLB flushes to complete +* which are not tracked with an explicit TLB fence, however at this +* stage that will never happen since the CT is already disabled, so +* make sure we signal them here under the assumption that we have +* completed a full GT reset. +*/ +WRITE_ONCE (gt->tlb_invalidation.seqno_recv, gt->tlb_invalidation.seqno); +wake_up_all (&guc->ct.wq); -mutex_lock (>->uc.guc.ct.lock);list_for_each_entry_safe (fence, next, >->tlb_invalidation.pending_fences, link)invalidation_fence_signal (fence);
Signature · 1 line
2.41.0
PATCH v5 6/10 drm/xe/ct: serialise fast_lock during CT disable
In reply to Matthew AuldThe fast-path CT could be running as we enter a runtime-suspend or potentially a GT reset, however here we only use the ct->fast_lock and not the full ct->lock. Before disabling the CT, also serialise against the fast_lock to ensure any in-progress work finishes before we start nuking the CT related stuff. Once we disable ct->enabled and drop the lock, any new work should fail gracefully, and anything that was in progress should be finished.
--- drivers/gpu/drm/xe/xe_guc_ct.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_guc_ct.c +6 -2
@@ -301,8 +301,10 @@ int xe_guc_ct_enable(struct xe_guc_ct *ct) goto err_out;mutex_lock (&ct->lock); +spin_lock_irq (&ct->fast_lock); ct->g2h_outstanding =0 ; ct->enabled = true; +spin_unlock_irq (&ct->fast_lock);mutex_unlock (&ct->lock);smp_mb ();
@@ -319,8 +321,10 @@ int xe_guc_ct_enable(struct xe_guc_ct *ct)void xe_guc_ct_disable (struct xe_guc_ct *ct) { -mutex_lock (&ct->lock); - ct->enabled = false; +mutex_lock (&ct->lock);/* Serialise dequeue_one_g2h() */ +spin_lock_irq (&ct->fast_lock);/* Serialise CT fast-path */ + ct->enabled = false;/* Finally disable CT communication */ +spin_unlock_irq (&ct->fast_lock);mutex_unlock (&ct->lock);xa_destroy (&ct->fence_lookup);
Signature · 1 line
2.41.0
PATCH v5 9/10 drm/xe: handle TLB invalidations from CT fast-path
In reply to Matthew AuldIn various test cases that put the system under a heavy load, we can sometimes see errors with missed TLB invalidations. In such cases we see the interrupt arrive for the invalidation from the GuC, however the actual processing of the completion is pushed onto a workqueue and handled with all the other CT stuff, which might take longer than expected. Since we expect TLB invalidations to complete within a reasonable amount of time (at most ~250ms), and they do seem pretty critical, allow handling directly from the CT fast-path.
v2 (José):
- Actually use the correct spinlock/unlock_irq, since pending_lock is grabbed from IRQ.
v3:
- Don't publish the TLB fence on the list until after we fully initialize it and successfully do the CT send. The list is now only protected by the spin_lock pending_lock and we can't hold that across the entire TLB send operation.
v4 (Matt Brost):
- Be careful with racing against fast CT path writing the seqno, before we have actually published the fence.
References: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/297
References: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/320
References: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/449
--- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c | 135 ++++++++++++-------- drivers/gpu/drm/xe/xe_gt_types.h | 5 + drivers/gpu/drm/xe/xe_guc_ct.c | 12 +- 3 files changed, 92 insertions(+), 60 deletions(-)
drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c +85 -50
@@ -25,7 +25,7 @@ static void xe_gt_tlb_fence_timeout(struct work_struct *work) tlb_invalidation.fence_tdr.work);struct xe_gt_tlb_invalidation_fence *fence, *next; -mutex_lock (>->uc.guc.ct.lock); +spin_lock_irq (>->tlb_invalidation.pending_lock);list_for_each_entry_safe (fence, next, >->tlb_invalidation.pending_fences, link) { s64 since_inval_ms = ktime_ms_delta(ktime_get(),
@@ -47,7 +47,7 @@ static void xe_gt_tlb_fence_timeout(struct work_struct *work)queue_delayed_work (system_wq, >->tlb_invalidation.fence_tdr,TLB_TIMEOUT ); -mutex_unlock (>->uc.guc.ct.lock); +spin_unlock_irq (>->tlb_invalidation.pending_lock); }/**
@@ -63,6 +63,7 @@ int xe_gt_tlb_invalidation_init(struct xe_gt *gt) { gt->tlb_invalidation.seqno =1 ;INIT_LIST_HEAD (>->tlb_invalidation.pending_fences); +spin_lock_init (>->tlb_invalidation.pending_lock);spin_lock_init (>->tlb_invalidation.lock); gt->tlb_invalidation.fence_context =dma_fence_context_alloc (1 ); INIT_DELAYED_WORK(>->tlb_invalidation.fence_tdr,
@@ -72,14 +73,20 @@ int xe_gt_tlb_invalidation_init(struct xe_gt *gt) }static void -invalidation_fence_signal (struct xe_gt_tlb_invalidation_fence *fence) +__invalidation_fence_signal (struct xe_gt_tlb_invalidation_fence *fence) {trace_xe_gt_tlb_invalidation_fence_signal (fence); -list_del (&fence->link);dma_fence_signal (&fence->base);dma_fence_put (&fence->base); } +static void +invalidation_fence_signal (struct xe_gt_tlb_invalidation_fence *fence) +{ +__invalidation_fence_signal (fence); +list_del (&fence->link); +} +/** * xe_gt_tlb_invalidation_reset - Initialize GT TLB invalidation reset * @gt: graphics tile
@@ -97,6 +104,7 @@ void xe_gt_tlb_invalidation_reset(struct xe_gt *gt)*/ mutex_lock (>->uc.guc.ct.lock); +spin_lock_irq (>->tlb_invalidation.pending_lock);cancel_delayed_work (>->tlb_invalidation.fence_tdr);/* * We might have various kworkers waiting for TLB flushes to complete
@@ -111,9 +119,23 @@ void xe_gt_tlb_invalidation_reset(struct xe_gt *gt)list_for_each_entry_safe (fence, next, >->tlb_invalidation.pending_fences, link)invalidation_fence_signal (fence); +spin_unlock_irq (>->tlb_invalidation.pending_lock);mutex_unlock (>->uc.guc.ct.lock); } +static bool tlb_invalidation_seqno_past (struct xe_gt *gt,int seqno) +{ +int seqno_recv =READ_ONCE (gt->tlb_invalidation.seqno_recv); + +if (seqno - seqno_recv < -(TLB_INVALIDATION_SEQNO_MAX /2 )) +return false; + +if (seqno - seqno_recv > (TLB_INVALIDATION_SEQNO_MAX /2 )) +return true; + +return seqno_recv >= seqno; +} +static int send_tlb_invalidation (struct xe_guc *guc,struct xe_gt_tlb_invalidation_fence *fence,u32 *action,int len)
@@ -121,7 +143,6 @@ static int send_tlb_invalidation(struct xe_guc *guc,struct xe_gt *gt =guc_to_gt (guc);int seqno;int ret; -bool queue_work;/* * XXX: The seqno algorithm relies on TLB invalidation being processed
@@ -132,21 +153,36 @@ static int send_tlb_invalidation(struct xe_guc *guc,mutex_lock (&guc->ct.lock); seqno = gt->tlb_invalidation.seqno;if (fence) { - queue_work =list_empty (>->tlb_invalidation.pending_fences); fence->seqno = seqno; -list_add_tail (&fence->link, - >->tlb_invalidation.pending_fences);trace_xe_gt_tlb_invalidation_fence_send (fence); } action[1 ] = seqno; ret =xe_guc_ct_send_locked (&guc->ct, action, len,G2H_LEN_DW_TLB_INVALIDATE ,1 );if (!ret && fence) { - fence->invalidation_time =ktime_get (); -if (queue_work) -queue_delayed_work (system_wq, - >->tlb_invalidation.fence_tdr, -TLB_TIMEOUT ); +spin_lock_irq (>->tlb_invalidation.pending_lock); +/* +* We haven't actually published the TLB fence as per +* pending_fences, but in theory our seqno could have already +* been written as we acquired the pending_lock. In such a case +* we can just go ahead and signal the fence here. +*/ +if (tlb_invalidation_seqno_past (gt, seqno)) { +__invalidation_fence_signal (fence); + }else { + fence->invalidation_time =ktime_get (); +list_add_tail (&fence->link, + >->tlb_invalidation.pending_fences); + +if (list_is_singular (>->tlb_invalidation.pending_fences)) +queue_delayed_work (system_wq, + >->tlb_invalidation.fence_tdr, +TLB_TIMEOUT ); + + } +spin_unlock_irq (>->tlb_invalidation.pending_lock); + }else if (ret <0 && fence) { +__invalidation_fence_signal (fence); }if (!ret) { gt->tlb_invalidation.seqno = (gt->tlb_invalidation.seqno + 1) %
@@ -155,8 +191,6 @@ static int send_tlb_invalidation(struct xe_guc *guc, gt->tlb_invalidation.seqno =1 ; ret = seqno; } -if (ret <0 && fence) -invalidation_fence_signal (fence);mutex_unlock (&guc->ct.lock);return ret;
@@ -271,19 +305,6 @@ int xe_gt_tlb_invalidation_vma(struct xe_gt *gt,return ret; } -static bool tlb_invalidation_seqno_past (struct xe_gt *gt,int seqno) -{ -int seqno_recv =READ_ONCE (gt->tlb_invalidation.seqno_recv); - -if (seqno - seqno_recv < -(TLB_INVALIDATION_SEQNO_MAX /2 )) -return false; - -if (seqno - seqno_recv > (TLB_INVALIDATION_SEQNO_MAX /2 )) -return true; - -return seqno_recv >= seqno; -} -/** * xe_gt_tlb_invalidation_wait - Wait for TLB to complete * @gt: graphics tile
@@ -331,22 +352,31 @@ int xe_gt_tlb_invalidation_wait(struct xe_gt *gt, int seqno)int xe_guc_tlb_invalidation_done_handler (struct xe_guc *guc,u32 *msg,u32 len) {struct xe_gt *gt =guc_to_gt (guc); -struct xe_gt_tlb_invalidation_fence *fence; -int expected_seqno; - -lockdep_assert_held (&guc->ct.lock); +struct xe_gt_tlb_invalidation_fence *fence, *next; +unsigned long flags;if (unlikely (len !=1 ))return -EPROTO ; -/* Sanity check on seqno */ - expected_seqno = (gt->tlb_invalidation.seqno_recv +1 ) % -TLB_INVALIDATION_SEQNO_MAX ; -if (!expected_seqno) - expected_seqno =1 ; -if (drm_WARN_ON (>_to_xe (gt)->drm, expected_seqno != msg[0 ])) { -drm_err (>_to_xe (gt)->drm,"TLB expected_seqno(%d) != msg(%u)\n" , - expected_seqno, msg[0 ]); +/* +* This can also be run both directly from the IRQ handler and also in +* process_g2h_msg(). Only one may process any individual CT message, +* however the order they are processed here could result in skipping a +* seqno. To handle that we just process all the seqnos from the last +* seqno_recv up to and including the one in msg[0]. The delta should be +* very small so there shouldn't be much of pending_fences we actually +* need to iterate over here. +* +* From GuC POV we expect the seqnos to always appear in-order, so if we +* see something later in the timeline we can be sure that anything +* appearing earlier has already signalled, just that we have yet to +* officially process the CT message like if racing against +* process_g2h_msg(). +*/ +spin_lock_irqsave (>->tlb_invalidation.pending_lock, flags); +if (tlb_invalidation_seqno_past (gt, msg[0 ])) { +spin_unlock_irqrestore (>->tlb_invalidation.pending_lock, flags); +return 0 ; }/*
@@ -356,19 +386,24 @@ int xe_guc_tlb_invalidation_done_handler(struct xe_guc *guc, u32 *msg, u32 len)WRITE_ONCE (gt->tlb_invalidation.seqno_recv, msg[0 ]);wake_up_all (&guc->ct.wq); - fence =list_first_entry_or_null (>->tlb_invalidation.pending_fences, -typeof (*fence), link); -if (fence) +list_for_each_entry_safe (fence, next, + >->tlb_invalidation.pending_fences, link) {trace_xe_gt_tlb_invalidation_fence_recv (fence); -if (fence &&tlb_invalidation_seqno_past (gt, fence->seqno)) { + +if (!tlb_invalidation_seqno_past (gt, fence->seqno)) +break ; +invalidation_fence_signal (fence); -if (!list_empty (>->tlb_invalidation.pending_fences)) -mod_delayed_work (system_wq, - >->tlb_invalidation.fence_tdr, -TLB_TIMEOUT ); -else -cancel_delayed_work (>->tlb_invalidation.fence_tdr); } +if (!list_empty (>->tlb_invalidation.pending_fences)) +mod_delayed_work (system_wq, + >->tlb_invalidation.fence_tdr, +TLB_TIMEOUT ); +else +cancel_delayed_work (>->tlb_invalidation.fence_tdr); + +spin_unlock_irqrestore (>->tlb_invalidation.pending_lock, flags); +return 0 ; }
drivers/gpu/drm/xe/xe_gt_types.h +5 -0
@@ -163,6 +163,11 @@ struct xe_gt {* invaliations, protected by CT lock */ struct list_head pending_fences; +/** +* @pending_lock: protects @pending_fences and updating +* @seqno_recv. +*/ +spinlock_t pending_lock;/** * @fence_tdr: schedules a delayed call to * xe_gt_tlb_fence_timeout after the timeut interval is over.
drivers/gpu/drm/xe/xe_guc_ct.c +2 -10
@@ -994,15 +994,8 @@ static int g2h_read(struct xe_guc_ct *ct, u32 *msg, bool fast_path)return 0 ;switch (FIELD_GET (GUC_HXG_EVENT_MSG_0_ACTION , msg[1 ])) { -/* -* FIXME: We really should process -* XE_GUC_ACTION_TLB_INVALIDATION_DONE here in the fast-path as -* these critical for page fault performance. We currently can't -* due to TLB invalidation done algorithm expecting the seqno -* returned in-order. With some small changes to the algorithm -* and locking we should be able to support out-of-order seqno. -*/ case XE_GUC_ACTION_REPORT_PAGE_FAULT_REQ_DESC : +case XE_GUC_ACTION_TLB_INVALIDATION_DONE :break ;/* Process these in fast-path */ default :return 0 ;
@@ -1056,8 +1049,7 @@ void xe_guc_ct_fast_path(struct xe_guc_ct *ct)struct xe_device *xe =ct_to_xe (ct);int len; -if (!xe_device_in_fault_mode (xe) || - !xe_device_mem_access_get_if_ongoing (xe)) +if (!xe_device_mem_access_get_if_ongoing (xe))return ;spin_lock (&ct->fast_lock);
Signature · 1 line
2.41.0
PATCH v5 10/10 drm/xe/tlb: print seqno_recv on fence TLB timeout
In reply to Matthew AuldTo help debugging, sample the current seqno_recv and dump it out if we encounter a TLB timeout for the fences path.
--- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c +2 -2
@@ -35,8 +35,8 @@ static void xe_gt_tlb_fence_timeout(struct work_struct *work)break ;trace_xe_gt_tlb_invalidation_fence_timeout (fence); -drm_err (>_to_xe (gt)->drm,"gt%d: TLB invalidation fence timeout, seqno=%d" , - gt->info.id, fence->seqno); +drm_err (>_to_xe (gt)->drm,"gt%d: TLB invalidation fence timeout, seqno=%d recv=%d" , + gt->info.id, fence->seqno, gt->tlb_invalidation.seqno_recv);list_del (&fence->link); fence->base.error = -ETIME ;
Signature · 1 line
2.41.0
Patchwork
CI[Intel-xe] ✓ CI.Patch_applied: success for Try to handle TLB invalidations from CT fast-path (rev3)
In reply to Matthew Auld== Series Details ==
== Series Details ==
Series: Try to handle TLB invalidations from CT fast-path (rev3)
URL : https://patchwork.freedesktop.org/series/120174/
State : success
== Summary ==
=== Applying kernel patches on branch 'drm-xe-next' with base: === Base commit: abcb88850 drm/xe: make kobject type struct as constant === git am output follows ===
Applying: drm/xe/tlb: drop unnecessary smp_wmb()
Applying: drm/xe/tlb: ensure we access seqno_recv once
Applying: drm/xe: hold mem_access.ref for CT fast-path
Applying: drm/xe/ct: hold fast_lock when reserving space for g2h
Applying: drm/xe/tlb: increment next seqno after successful CT send
Applying: drm/xe/ct: serialise fast_lock during CT disable
Applying: drm/xe/gt: tweak placement for signalling TLB fences after GT reset
Applying: drm/xe/tlb: also update seqno_recv during reset
Applying: drm/xe: handle TLB invalidations from CT fast-path
Applying: drm/xe/tlb: print seqno_recv on fence TLB timeout
Patchwork
CI[Intel-xe] ✗ CI.checkpatch: warning for Try to handle TLB invalidations from CT fast-path (rev3)
In reply to Matthew Auld== Series Details ==
== Series Details ==
Series: Try to handle TLB invalidations from CT fast-path (rev3)
URL : https://patchwork.freedesktop.org/series/120174/
State : warning
== Summary ==
+ KERNEL=/kernel
+ git clone https://gitlab.freedesktop.org/drm/maintainer-tools mt Cloning into 'mt'...
warning: redirecting to https://gitlab.freedesktop.org/drm/maintainer-tools.git/
+ git -C mt rev-list -n1 origin/master c7d32770e3cd31d9fc134ce41f329b10aa33ee15
+ cd /kernel
+ git config --global --add safe.directory /kernel
+ git log -n1
commit acf7079ed87bae407c14c047401f2eab92c7681f
Author: Matthew Auld <[email protected]> Date: Thu Jul 6 17:20:14 2023 +0100
drm/xe/tlb: print seqno_recv on fence TLB timeout
To help debugging, sample the current seqno_recv and dump it out if we encounter a TLB timeout for the fences path.
+ /mt/dim checkpatch abcb88850f885254cf7477e1f7a5956136fb547c drm-intel506d38469 drm/xe/tlb: drop unnecessary smp_wmb()48c1fe486 drm/xe/tlb: ensure we access seqno_recv oncef79cf87dc drm/xe: hold mem_access.ref for CT fast-path0905214a5 drm/xe/ct: hold fast_lock when reserving space for g2hbf675b585 drm/xe/tlb: increment next seqno after successful CT sendfd34513ed drm/xe/ct: serialise fast_lock during CT disable7216e6fb2 drm/xe/gt: tweak placement for signalling TLB fences after GT reset9054a087b drm/xe/tlb: also update seqno_recv during reset1b74b633d drm/xe: handle TLB invalidations from CT fast-path
-:30: WARNING:COMMIT_LOG_USE_LINK: Unknown link reference 'References:', use 'Link:' or 'Closes:' instead #30:
References: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/297
-:31: WARNING:COMMIT_LOG_USE_LINK: Unknown link reference 'References:', use 'Link:' or 'Closes:' instead #31:
References: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/320
-:32: WARNING:COMMIT_LOG_USE_LINK: Unknown link reference 'References:', use 'Link:' or 'Closes:' instead #32:
References: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/449
-:169: CHECK:BRACES: Blank lines aren't necessary before a close brace '}' #169: FILE: drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c:182:
+
+ }
total: 0 errors, 3 warnings, 1 checks, 266 lines checked acf7079ed drm/xe/tlb: print seqno_recv on fence TLB timeout
Re: [Intel-xe] [PATCH v5 01/10] drm/xe/tlb: drop unnecessary smp_wmb()
In reply to Matthew AuldQuoted 10 lines · Matthew Auld
wake_up_all()andwait_event_timeout()already have the correct barriers as per https://www.kernel.org/doc/Documentation/memory-barriers.txt.
This should ensure that theseqno_recvwrite can't be re-ordered wrt to the actualwake_up_all()i.e we get woken up but there is no write. The reader side withwait_event_timeout()also has the correct barriers. With that drop the hand rolledsmp_wmb(), which is anyway missing some kind of matching barrier on the reader side.Signed-off-by Matthew Auld <[email protected]>Cc Matthew Brost <[email protected]>
Quoted 25 lines
Cc José Roberto de Souza <[email protected]>--- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c +4
-1@@ -333,8 +333,11 @@ int xe_guc_tlb_invalidation_done_handler(struct xe_guc *guc, u32 *msg, u32 len) expected_seqno, msg[0]); } +/* +* wake_up_all() and wait_event_timeout() already have the correct +* barriers. +*/ gt->tlb_invalidation.seqno_recv = msg[0 ]; -smp_wmb ();wake_up_all (&guc->ct.wq); fence = list_first_entry_or_null(>->tlb_invalidation.pending_fences,Signature · 1 line
2.41.0
Re: [Intel-xe] [PATCH v5 02/10] drm/xe/tlb: ensure we access seqno_recv once
In reply to Matthew AuldQuoted 8 lines · Matthew Auld
Ensure we load
gt->tlb_invalidation.seqno_recvonce, and use that for our seqno checking. Thegt->tlb_invalidation_seqno_pastis a shared global variable and can potentially change at any point here. However the checks here need to operate on a stable version ofseqno_recvfor this to make any sense.Signed-off-by Matthew Auld <[email protected]>Cc Matthew Brost <[email protected]>
Quoted 42 lines
Cc José Roberto de Souza <[email protected]>--- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c +6
-6@@ -257,15 +257,15 @@ int xe_gt_tlb_invalidation_vma(struct xe_gt *gt,static bool tlb_invalidation_seqno_past (struct xe_gt *gt,int seqno) { -if (seqno - gt->tlb_invalidation.seqno_recv < - -(TLB_INVALIDATION_SEQNO_MAX /2 )) +int seqno_recv =READ_ONCE (gt->tlb_invalidation.seqno_recv); + +if (seqno - seqno_recv < -(TLB_INVALIDATION_SEQNO_MAX /2 ))return false; -if (seqno - gt->tlb_invalidation.seqno_recv > - (TLB_INVALIDATION_SEQNO_MAX /2 )) +if (seqno - seqno_recv > (TLB_INVALIDATION_SEQNO_MAX /2 ))return true; -return gt->tlb_invalidation.seqno_recv >= seqno; +return seqno_recv >= seqno; }/** @@ -337,7 +337,7 @@ int xe_guc_tlb_invalidation_done_handler(struct xe_guc *guc, u32 *msg, u32 len)* wake_up_all() and wait_event_timeout() already have the correct * barriers. */ - gt->tlb_invalidation.seqno_recv = msg[0 ]; +WRITE_ONCE (gt->tlb_invalidation.seqno_recv, msg[0 ]);wake_up_all (&guc->ct.wq); fence = list_first_entry_or_null(>->tlb_invalidation.pending_fences,Signature · 1 line
2.41.0
Re: [Intel-xe] [PATCH v5 10/10] drm/xe/tlb: print seqno_recv on fence TLB timeout
In reply to Matthew AuldQuoted 5 lines · Matthew Auld
To help debugging, sample the current
seqno_recvand dump it out if we encounter a TLB timeout for the fences path.Signed-off-by Matthew Auld <[email protected]>Cc Matthew Brost <[email protected]>
Quoted 23 lines
Cc José Roberto de Souza <[email protected]>--- drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c +2
-2@@ -35,8 +35,8 @@ static void xe_gt_tlb_fence_timeout(struct work_struct *work)break ;trace_xe_gt_tlb_invalidation_fence_timeout (fence); -drm_err (>_to_xe (gt)->drm,"gt%d: TLB invalidation fence timeout, seqno=%d" , - gt->info.id, fence->seqno); +drm_err (>_to_xe (gt)->drm,"gt%d: TLB invalidation fence timeout, seqno=%d recv=%d" , + gt->info.id, fence->seqno, gt->tlb_invalidation.seqno_recv);list_del (&fence->link); fence->base.error = -ETIME ;Signature · 1 line
2.41.0
Quoted 4 lines · Matthew Auld
In various test cases and workloads that put the system under a heavy load, we can sometimes see errors with missed TLB invalidations. If we handle the TLB completions directly from IRQ we can see a big improvement.
Quoted 8 lines
v4:
- Various tweaks. Big thing is to make sure we grab thefast_lockfor the g2h reservation when doing the CT send, which closes one big race with the CT fast-path.
v5:
- Various improvements. Fix the race with adding a fence after its seqno has already been signalled. (Matt Brost)
Regressions fixed in this version.
Still getting some TLB timeouts like this one(https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/289) but this series reduced the number of TLB timeouts.