Skip to contentSkip to conversation

drm/scheduler: set entity to NULL in drm_sched_entity_pop_job()

PATCH Superseded Danilo Krummrich 3 participants 6 messages Mar 31, 2023 – Apr 11, 2023

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/scheduler: set entity to NULL in drm_sched_entity_pop_job()

It already happend a few times that patches slipped through which implemented access to an entity through a job that was already removed from the entities queue. Since jobs and entities might have different lifecycles, this can potentially cause UAF bugs.

In order to make it obvious that a jobs entity pointer shouldn't be accessed after drm_sched_entity_pop_job() was called successfully, set the jobs entity pointer to NULL once the job is removed from the entity queue.

Moreover, debugging a potential NULL pointer dereference is way easier than potentially corrupted memory through a UAF.

---
I'm aware that drivers could already use job->entity in arbitrary places, since they in control of when the entity is actually freed. A quick grep didn't give me any results where this would actually be the case, however maybe I also just didn't catch it.

If, therefore, we don't want to set job->entity to NULL I think we should at least add a comment somewhere.
---

 drivers/gpu/drm/scheduler/sched_entity.c | 6 ++++++
 1 file changed, 6 insertions(+)
drivers/gpu/drm/scheduler/sched_entity.c +6 -0
index 15d04a0ec623..a9c6118e534b 100644
@@ -448,6 +448,12 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
 			drm_sched_rq_update_fifo(entity, next->submit_ts);
 	}
 
+	/* Jobs and entities might have different lifecycles. Since we're
+	 * removing the job from the entities queue, set the jobs entity pointer
+	 * to NULL to prevent any future access of the entity through this job.
+	 */
+	sched_job->entity = NULL;
+
 	return sched_job;
 }
 
Signature · 1 line
2.39.2
#1 lore raw

Am 31.03.23 um 02:06 schrieb Danilo Krummrich:

Quoted 10 lines

It already happend a few times that patches slipped through which implemented access to an entity through a job that was already removed from the entities queue. Since jobs and entities might have different lifecycles, this can potentially cause UAF bugs.

In order to make it obvious that a jobs entity pointer shouldn't be accessed after drm_sched_entity_pop_job() was called successfully, set the jobs entity pointer to NULL once the job is removed from the entity queue.

Quoted 4 lines

Moreover, debugging a potential NULL pointer dereference is way easier than potentially corrupted memory through a UAF.

In general "YES PLEASE!", but I fear that this will break amdgpus reset sequence.

On the other hand when amdgpu still relies on that pointer it's clearly a bug (which I pointed out tons of times before).

Luben any opinion on that? Could you drive cleaning that up as well?

Thanks,
Christian.

Quoted 30 lines

---
I'm aware that drivers could already use job->entity in arbitrary places, since they in control of when the entity is actually freed. A quick grep didn't give me any results where this would actually be the case, however maybe I also just didn't catch it.

If, therefore, we don't want to set job->entity to NULL I think we should at least add a comment somewhere.
---

  drivers/gpu/drm/scheduler/sched_entity.c | 6 ++++++
  1 file changed, 6 insertions(+)
drivers/gpu/drm/scheduler/sched_entity.c +6 -0
index 15d04a0ec623..a9c6118e534b 100644
@@ -448,6 +448,12 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
  			drm_sched_rq_update_fifo(entity, next->submit_ts);
  	}
 
+	/* Jobs and entities might have different lifecycles. Since we're
+	 * removing the job from the entities queue, set the jobs entity pointer
+	 * to NULL to prevent any future access of the entity through this job.
+	 */
+	sched_job->entity = NULL;
+
  	return sched_job;
  }
 
#2 lore raw
Quoted 16 lines · Christian König

Am 31.03.23 um 02:06 schrieb Danilo Krummrich:

Quoted 10 lines

It already happend a few times that patches slipped through which implemented access to an entity through a job that was already removed from the entities queue. Since jobs and entities might have different lifecycles, this can potentially cause UAF bugs.

In order to make it obvious that a jobs entity pointer shouldn't be accessed after drm_sched_entity_pop_job() was called successfully, set the jobs entity pointer to NULL once the job is removed from the entity queue.

Quoted 4 lines

Moreover, debugging a potential NULL pointer dereference is way easier than potentially corrupted memory through a UAF.

Quoted 7 lines

In general "YES PLEASE!", but I fear that this will break amdgpus reset sequence.

On the other hand when amdgpu still relies on that pointer it's clearly a bug (which I pointed out tons of times before).

Luben any opinion on that? Could you drive cleaning that up as well?

Hi Christian,

No worries, yes, I'll take a look at this after breakfast.

Quoted 12 lines

Thanks,
Christian.

Quoted 8 lines

---
I'm aware that drivers could already use job->entity in arbitrary places, since they in control of when the entity is actually freed. A quick grep didn't give me any results where this would actually be the case, however maybe I also just didn't catch it.

If, therefore, we don't want to set job->entity to NULL I think we should at least add a comment somewhere.

I agree with the sentiment of this patch. I'll have to take a closer look at this because there was some indirect pointer dependency due to the way the FIFO was implemented, and I review the code every 3-6 months to remind me of that--maybe it's related, maybe not. But this looks like a something we can delve into and at best come up with a comment explaining what's going on and why.

We haven't seen any oopses so far the way this is, and any new patches which evoke an oops, may be doing something they shouldn't. I'll take a look.

Any indication of what these new patches were doing?

Regards,
Luben

Quoted 23 lines
Quoted 22 lines

---

  drivers/gpu/drm/scheduler/sched_entity.c | 6 ++++++
  1 file changed, 6 insertions(+)
drivers/gpu/drm/scheduler/sched_entity.c +6 -0
index 15d04a0ec623..a9c6118e534b 100644
@@ -448,6 +448,12 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
  			drm_sched_rq_update_fifo(entity, next->submit_ts);
  	}
 
+	/* Jobs and entities might have different lifecycles. Since we're
+	 * removing the job from the entities queue, set the jobs entity pointer
+	 * to NULL to prevent any future access of the entity through this job.
+	 */
+	sched_job->entity = NULL;
+
  	return sched_job;
  }
 
#3 lore raw
In reply to Christian König
Quoted 16 lines · Christian König

Am 31.03.23 um 02:06 schrieb Danilo Krummrich:

Quoted 10 lines

It already happend a few times that patches slipped through which implemented access to an entity through a job that was already removed from the entities queue. Since jobs and entities might have different lifecycles, this can potentially cause UAF bugs.

In order to make it obvious that a jobs entity pointer shouldn't be accessed after drm_sched_entity_pop_job() was called successfully, set the jobs entity pointer to NULL once the job is removed from the entity queue.

Quoted 4 lines

Moreover, debugging a potential NULL pointer dereference is way easier than potentially corrupted memory through a UAF.

Quoted 7 lines

In general "YES PLEASE!", but I fear that this will break amdgpus reset sequence.

On the other hand when amdgpu still relies on that pointer it's clearly a bug (which I pointed out tons of times before).

Luben any opinion on that? Could you drive cleaning that up as well?

I didn't find any references to scheduling entity after the job is submitted to the hardware. (I commented the same in the other thread, we just need to decide which way to go.)

Regards,
Luben

Quoted 35 lines

Thanks,
Christian.

Quoted 30 lines

---
I'm aware that drivers could already use job->entity in arbitrary places, since they in control of when the entity is actually freed. A quick grep didn't give me any results where this would actually be the case, however maybe I also just didn't catch it.

If, therefore, we don't want to set job->entity to NULL I think we should at least add a comment somewhere.
---

  drivers/gpu/drm/scheduler/sched_entity.c | 6 ++++++
  1 file changed, 6 insertions(+)
drivers/gpu/drm/scheduler/sched_entity.c +6 -0
index 15d04a0ec623..a9c6118e534b 100644
@@ -448,6 +448,12 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
  			drm_sched_rq_update_fifo(entity, next->submit_ts);
  	}
 
+	/* Jobs and entities might have different lifecycles. Since we're
+	 * removing the job from the entities queue, set the jobs entity pointer
+	 * to NULL to prevent any future access of the entity through this job.
+	 */
+	sched_job->entity = NULL;
+
  	return sched_job;
  }
 
#4 lore raw
Quoted 20 lines · Luben Tuikov
Quoted 19 lines · Christian König

Am 31.03.23 um 02:06 schrieb Danilo Krummrich:

Quoted 10 lines

It already happend a few times that patches slipped through which implemented access to an entity through a job that was already removed from the entities queue. Since jobs and entities might have different lifecycles, this can potentially cause UAF bugs.

In order to make it obvious that a jobs entity pointer shouldn't be accessed after drm_sched_entity_pop_job() was called successfully, set the jobs entity pointer to NULL once the job is removed from the entity queue.

Quoted 4 lines

Moreover, debugging a potential NULL pointer dereference is way easier than potentially corrupted memory through a UAF.

In general "YES PLEASE!", but I fear that this will break amdgpus reset sequence.

Quoted 8 lines
Quoted 4 lines

On the other hand when amdgpu still relies on that pointer it's clearly a bug (which I pointed out tons of times before).

Luben any opinion on that? Could you drive cleaning that up as well?

I didn't find any references to scheduling entity after the job is submitted to the hardware. (I commented the same in the other thread, we just need to decide which way to go.)

AFAICS from the other mail thread it seems to be consensus to not ref-count entities and handle job statistics differently.

Should we go ahead and take this patch then? Maybe it also makes sense to send a V2 additionally adding a comment to the drm_sched_job structure mentioning that .entity must not be used after the job was taken from the entities queue.

- Danilo

Quoted 40 lines

Regards,
Luben

Quoted 35 lines

Thanks,
Christian.

Quoted 30 lines

---
I'm aware that drivers could already use job->entity in arbitrary places, since they in control of when the entity is actually freed. A quick grep didn't give me any results where this would actually be the case, however maybe I also just didn't catch it.

If, therefore, we don't want to set job->entity to NULL I think we should at least add a comment somewhere.
---

   drivers/gpu/drm/scheduler/sched_entity.c | 6 ++++++
   1 file changed, 6 insertions(+)
drivers/gpu/drm/scheduler/sched_entity.c +6 -0
index 15d04a0ec623..a9c6118e534b 100644
@@ -448,6 +448,12 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity)
   			drm_sched_rq_update_fifo(entity, next->submit_ts);
   	}
 
+	/* Jobs and entities might have different lifecycles. Since we're
+	 * removing the job from the entities queue, set the jobs entity pointer
+	 * to NULL to prevent any future access of the entity through this job.
+	 */
+	sched_job->entity = NULL;
+
   	return sched_job;
   }
 
#5 lore raw
Quoted 30 lines · Danilo Krummrich
Quoted 20 lines · Luben Tuikov
Quoted 19 lines · Christian König

Am 31.03.23 um 02:06 schrieb Danilo Krummrich:

Quoted 10 lines

It already happend a few times that patches slipped through which implemented access to an entity through a job that was already removed from the entities queue. Since jobs and entities might have different lifecycles, this can potentially cause UAF bugs.

In order to make it obvious that a jobs entity pointer shouldn't be accessed after drm_sched_entity_pop_job() was called successfully, set the jobs entity pointer to NULL once the job is removed from the entity queue.

Quoted 4 lines

Moreover, debugging a potential NULL pointer dereference is way easier than potentially corrupted memory through a UAF.

In general "YES PLEASE!", but I fear that this will break amdgpus reset sequence.

Quoted 8 lines
Quoted 4 lines

On the other hand when amdgpu still relies on that pointer it's clearly a bug (which I pointed out tons of times before).

Luben any opinion on that? Could you drive cleaning that up as well?

I didn't find any references to scheduling entity after the job is submitted to the hardware. (I commented the same in the other thread, we just need to decide which way to go.)

Quoted 7 lines

AFAICS from the other mail thread it seems to be consensus to not ref-count entities and handle job statistics differently.

Should we go ahead and take this patch then? Maybe it also makes sense to send a V2 additionally adding a comment to the drm_sched_job structure mentioning that .entity must not be used after the job was taken from the entities queue.

Yes, please send a v2, but instead of mentioning (or in addition to) that the job was taken from the "entities queue", mention that once the job is pushed to the hardware, i.e. the pending queue, from then on, the "entity" should not be referenced anymore. IOW, we want to mention that it is going from X to Y, as opposed to just that it's taken from X, because the latter begs the question "Where is it going to then?".

For the record, I think that using kref would give us the most stability, even if we skipped "entity" and let the scheduler, or even the controller keep a kref on submitted commands down to the GPU. On reset, we block command submission to the GPU from the outermost boundary, and then start peeling the layers from the innermost boundary.

Using kref also forces us to think objectively and set explicit (clear) dependencies from the outset--i.e. who gets freed and when and under what circumstances.

And using kref might even expose the wrong dependencies, thus prompting a redesign and thus a fix.

Signature · 2 lines
Regards,
Luben