Skip to contentSkip to conversation

drm/amdgpu: Make KIQ read/write register routine be atomic

PATCH In review Shaoyun Liu 4 participants 8 messages Jun 29, 2017 – Jul 5, 2017 7 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/amdgpu: Make KIQ read/write register routine be atomic

1. Use spin lock instead of mutex in KIQ
2. Directly write to KIQ fence address instead of using fence_emit()
3. Disable the interrupt for KIQ read/write and use CPU polling

Change-Id Id3693a2878ce1338f55aee3def6e7fc0f6b81996
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h      |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c  |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c | 57 ++++++++++++++++++++++----------
 3 files changed, 41 insertions(+), 20 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -1
index ab1dad2..a155206 100644
@@ -955,7 +955,7 @@ struct amdgpu_mec {
 struct amdgpu_kiq {
 	u64			eop_gpu_addr;
 	struct amdgpu_bo	*eop_obj;
-	struct mutex		ring_mutex;
+	spinlock_t		ring_lock;
 	struct amdgpu_ring	ring;
 	struct amdgpu_irq_src	irq;
 };
drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c +1 -1
index e26108a..e5e5541 100644
@@ -184,7 +184,7 @@ int amdgpu_gfx_kiq_init_ring(struct amdgpu_device *adev,
 	struct amdgpu_kiq *kiq = &adev->gfx.kiq;
 	int r = 0;
 
-	mutex_init(&kiq->ring_mutex);
+	spin_lock_init(&kiq->ring_lock);
 
 	r = amdgpu_wb_get(adev, &adev->virt.reg_val_offs);
 	if (r)
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c +39 -18

86 lines not on this page · raw message

Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1498770496-28032-1-git-send-email-Shaoyun.Liu-5C7GfCeVMHo@public.gmane.org

#2 lore raw
Quoted 3 lines · Shaoyun Liu

1. Use spin lock instead of mutex in KIQ
2. Directly write to KIQ fence address instead of using fence_emit()
3. Disable the interrupt for KIQ read/write and use CPU polling

This list indicates that this patch should be split up in at least three patches. :)

Signature · 6 lines
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 14d97048-a0ba-69cd-1eac-e82c4462c51f-otUistvHUpPR7s880joybQ@public.gmane.org

#4 lore raw

Am 30.06.2017 um 03:21 schrieb Michel Dänzer:

Quoted 6 lines · Shaoyun Liu
Quoted 3 lines

1. Use spin lock instead of mutex in KIQ
2. Directly write to KIQ fence address instead of using fence_emit()
3. Disable the interrupt for KIQ read/write and use CPU polling

This list indicates that this patch should be split up in at least three patches. :)

Yeah, apart from that is is not a good idea to mess with the fence internals directly in the KIQ code, please add a helper in the fence code for this.

Regards,
Christian.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): d9834700-f977-e5d5-2c79-7c2332ba44e9-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

#6 lore raw

Hi , Christian
The new code actually will not use the fence function , it just need a memory that expose both CPU and GPU address . Do you really want to add the wrap functions that just expose the CPU and GPU address in this case ?

Regards
Shaoyun.liu

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: Friday, June 30, 2017 3:57 AM
To: Michel Dänzer; Liu, Shaoyun
Cc: [email protected]
Subject: Re: [PATCH] drm/amdgpu: Make KIQ read/write register routine be atomic

Am 30.06.2017 um 03:21 schrieb Michel Dänzer:

Quoted 6 lines · Shaoyun Liu
Quoted 3 lines

1. Use spin lock instead of mutex in KIQ 2. Directly write to KIQ fence address instead of using fence_emit() 3. Disable the interrupt for KIQ read/write and use CPU polling

This list indicates that this patch should be split up in at least three patches. :)

Yeah, apart from that is is not a good idea to mess with the fence internals directly in the KIQ code, please add a helper in the fence code for this.

Regards,
Christian.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): DM5PR12MB11795DAAD88232D2874CEF17F4D30-2J9CzHegvk8p/[email protected]

#8 lore raw

Hi Shaoyun

looks you want to make KIQ reg access be atomic & UN-interruptible,

I think most user of KIQ reg access is not in atomic context, so your patch only benefit for the place using KIQ from IRQ.

why not implement another KIQ reg access function ? e.g. :

amdgpu_virt_kiq_rreg_atomic(...);
amdgpu_virt_kiq_wreg_atomic(...);

that way you can satisfy your requirement and not introduce unknown issue for other places that calling original virt_kiq_r/weg() function.

because busy polling have chance to hang cpu in SR-IOV (imagine 16 VF, and many VF doing FLR one by one, your busy polling may let guest kernel cpu stuck in ATOMIC_CONTEXT for more than 5 seconds )

BR Monk

________________________________
From: amd-gfx <amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> on behalf of Liu, Shaoyun <[email protected]>
Sent: Friday, June 30, 2017 10:55:39 PM
To: Christian König; Michel Dänzer
Cc: [email protected]
Subject: RE: [PATCH] drm/amdgpu: Make KIQ read/write register routine be atomic

Hi , Christian
The new code actually will not use the fence function , it just need a memory that expose both CPU and GPU address . Do you really want to add the wrap functions that just expose the CPU and GPU address in this case ?

Regards
Shaoyun.liu

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: Friday, June 30, 2017 3:57 AM
To: Michel Dänzer; Liu, Shaoyun
Cc: [email protected]
Subject: Re: [PATCH] drm/amdgpu: Make KIQ read/write register routine be atomic

Am 30.06.2017 um 03:21 schrieb Michel Dänzer:

Quoted 6 lines · Shaoyun Liu
Quoted 3 lines

1. Use spin lock instead of mutex in KIQ 2. Directly write to KIQ fence address instead of using fence_emit() 3. Disable the interrupt for KIQ read/write and use CPU polling

This list indicates that this patch should be split up in at least three patches. :)

Yeah, apart from that is is not a good idea to mess with the fence internals directly in the KIQ code, please add a helper in the fence code for this.

Regards,
Christian.
_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): BLUPR12MB04498F84A6D9E542753D234C84D60-7LeqcoF/[email protected]

#10 lore raw

Hi , Monk
The TLB flush code is not used from IRQ , but it's been called while holding spin_lock . This is the reason why I want to change the routine be atomic. After check the register spec again , I found that the TLB invalidation and HDP coherency cntl register are safe to use in the VFs and in current code we already try to avoid use KIQ in TLB flush function by directly program the TLB invalidate registers but just missed one place on HDP flush . So I think it 's better to fix the HDP flush code and leave current KIQ code un-touched .

Regards
Shaoyun.liu

From: Liu, Monk
Sent: Monday, July 03, 2017 5:45 AM
To: Liu, Shaoyun; Christian König; Michel Dänzer
Cc: [email protected]
Subject: Re: [PATCH] drm/amdgpu: Make KIQ read/write register routine be atomic

Hi Shaoyun

looks you want to make KIQ reg access be atomic & UN-interruptible,

I think most user of KIQ reg access is not in atomic context, so your patch only benefit for the place using KIQ from IRQ.

why not implement another KIQ reg access function ? e.g. :

amdgpu_virt_kiq_rreg_atomic(...);

amdgpu_virt_kiq_wreg_atomic(...);

that way you can satisfy your requirement and not introduce unknown issue for other places that calling original virt_kiq_r/weg() function.

because busy polling have chance to hang cpu in SR-IOV (imagine 16 VF, and many VF doing FLR one by one, your busy polling may let guest kernel cpu stuck in ATOMIC_CONTEXT for more than 5 seconds )

BR Monk

________________________________
From: amd-gfx <amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org<mailto:[email protected]>> on behalf of Liu, Shaoyun <[email protected]<mailto:[email protected]>>
Sent: Friday, June 30, 2017 10:55:39 PM
To: Christian König; Michel Dänzer
Cc: [email protected]<mailto:[email protected]>
Subject: RE: [PATCH] drm/amdgpu: Make KIQ read/write register routine be atomic

Hi , Christian
The new code actually will not use the fence function , it just need a memory that expose both CPU and GPU address . Do you really want to add the wrap functions that just expose the CPU and GPU address in this case ?

Regards
Shaoyun.liu

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: Friday, June 30, 2017 3:57 AM
To: Michel Dänzer; Liu, Shaoyun
Cc: [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Make KIQ read/write register routine be atomic

Am 30.06.2017 um 03:21 schrieb Michel Dänzer:

Quoted 6 lines · Shaoyun Liu
Quoted 3 lines

1. Use spin lock instead of mutex in KIQ 2. Directly write to KIQ fence address instead of using fence_emit() 3. Disable the interrupt for KIQ read/write and use CPU polling

This list indicates that this patch should be split up in at least three patches. :)

Yeah, apart from that is is not a good idea to mess with the fence internals directly in the KIQ code, please add a helper in the fence code for this.

Regards,
Christian.
_______________________________________________
amd-gfx mailing list [email protected]<mailto:[email protected]>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): DM5PR12MB11793F1D1012089CED2530C6F4D70-2J9CzHegvk8p/[email protected]

#12 lore raw

What do you mean HDP flush code ?

Can you elaborate it ? the only HDP flush I remember via CPU is the one in GMC_V9's gpu_tlb_flush, I saw it is still using regular register access, but I agree change it to CPU direct access

BR Monk

From: Liu, Shaoyun
Sent: Tuesday, July 04, 2017 2:14 PM
To: Liu, Monk <[email protected]>; Christian König <[email protected]>; Michel Dänzer <[email protected]>
Cc: [email protected]
Subject: RE: [PATCH] drm/amdgpu: Make KIQ read/write register routine be atomic

Hi , Monk
The TLB flush code is not used from IRQ , but it's been called while holding spin_lock . This is the reason why I want to change the routine be atomic. After check the register spec again , I found that the TLB invalidation and HDP coherency cntl register are safe to use in the VFs and in current code we already try to avoid use KIQ in TLB flush function by directly program the TLB invalidate registers but just missed one place on HDP flush . So I think it 's better to fix the HDP flush code and leave current KIQ code un-touched .

Regards
Shaoyun.liu

From: Liu, Monk
Sent: Monday, July 03, 2017 5:45 AM
To: Liu, Shaoyun; Christian König; Michel Dänzer
Cc: [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Make KIQ read/write register routine be atomic

Hi Shaoyun

looks you want to make KIQ reg access be atomic & UN-interruptible,

I think most user of KIQ reg access is not in atomic context, so your patch only benefit for the place using KIQ from IRQ.

why not implement another KIQ reg access function ? e.g. :

amdgpu_virt_kiq_rreg_atomic(...);
amdgpu_virt_kiq_wreg_atomic(...);

that way you can satisfy your requirement and not introduce unknown issue for other places that calling original virt_kiq_r/weg() function.

because busy polling have chance to hang cpu in SR-IOV (imagine 16 VF, and many VF doing FLR one by one, your busy polling may let guest kernel cpu stuck in ATOMIC_CONTEXT for more than 5 seconds )

BR Monk

________________________________
From: amd-gfx <amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org<mailto:[email protected]>> on behalf of Liu, Shaoyun <[email protected]<mailto:[email protected]>>
Sent: Friday, June 30, 2017 10:55:39 PM
To: Christian König; Michel Dänzer
Cc: [email protected]<mailto:[email protected]>
Subject: RE: [PATCH] drm/amdgpu: Make KIQ read/write register routine be atomic

Hi , Christian
The new code actually will not use the fence function , it just need a memory that expose both CPU and GPU address . Do you really want to add the wrap functions that just expose the CPU and GPU address in this case ?

Regards
Shaoyun.liu

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: Friday, June 30, 2017 3:57 AM
To: Michel Dänzer; Liu, Shaoyun
Cc: [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Make KIQ read/write register routine be atomic

Am 30.06.2017 um 03:21 schrieb Michel Dänzer:

Quoted 6 lines · Shaoyun Liu
Quoted 3 lines

1. Use spin lock instead of mutex in KIQ 2. Directly write to KIQ fence address instead of using fence_emit() 3. Disable the interrupt for KIQ read/write and use CPU polling

This list indicates that this patch should be split up in at least three patches. :)

Yeah, apart from that is is not a good idea to mess with the fence internals directly in the KIQ code, please add a helper in the fence code for this.

Regards,
Christian.
_______________________________________________
amd-gfx mailing list [email protected]<mailto:[email protected]>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): BLUPR12MB04491EEAF96DF02542047DA584D40-7LeqcoF/[email protected]

#14 lore raw

It's inside the gmc_v9_0_gart_flush_gpu_tlb function where it calls the nbio_v6_0_hdp_flush . I already sent out the commit for review .you can check for details .

Regards
Shaoyun.liu

From: Liu, Monk
Sent: Tuesday, July 04, 2017 11:26 PM
To: Liu, Shaoyun; Christian König; Michel Dänzer
Cc: [email protected]
Subject: RE: [PATCH] drm/amdgpu: Make KIQ read/write register routine be atomic

What do you mean HDP flush code ?

Can you elaborate it ? the only HDP flush I remember via CPU is the one in GMC_V9's gpu_tlb_flush, I saw it is still using regular register access, but I agree change it to CPU direct access

BR Monk

From: Liu, Shaoyun
Sent: Tuesday, July 04, 2017 2:14 PM
To: Liu, Monk <[email protected]<mailto:[email protected]>>; Christian König <[email protected]<mailto:[email protected]>>; Michel Dänzer <[email protected]<mailto:[email protected]>>
Cc: [email protected]<mailto:[email protected]>
Subject: RE: [PATCH] drm/amdgpu: Make KIQ read/write register routine be atomic

Hi , Monk
The TLB flush code is not used from IRQ , but it's been called while holding spin_lock . This is the reason why I want to change the routine be atomic. After check the register spec again , I found that the TLB invalidation and HDP coherency cntl register are safe to use in the VFs and in current code we already try to avoid use KIQ in TLB flush function by directly program the TLB invalidate registers but just missed one place on HDP flush . So I think it 's better to fix the HDP flush code and leave current KIQ code un-touched .

Regards
Shaoyun.liu

From: Liu, Monk
Sent: Monday, July 03, 2017 5:45 AM
To: Liu, Shaoyun; Christian König; Michel Dänzer
Cc: [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Make KIQ read/write register routine be atomic

Hi Shaoyun

looks you want to make KIQ reg access be atomic & UN-interruptible,

I think most user of KIQ reg access is not in atomic context, so your patch only benefit for the place using KIQ from IRQ.

why not implement another KIQ reg access function ? e.g. :

amdgpu_virt_kiq_rreg_atomic(...);
amdgpu_virt_kiq_wreg_atomic(...);

that way you can satisfy your requirement and not introduce unknown issue for other places that calling original virt_kiq_r/weg() function.

because busy polling have chance to hang cpu in SR-IOV (imagine 16 VF, and many VF doing FLR one by one, your busy polling may let guest kernel cpu stuck in ATOMIC_CONTEXT for more than 5 seconds )

BR Monk

________________________________
From: amd-gfx <amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org<mailto:[email protected]>> on behalf of Liu, Shaoyun <[email protected]<mailto:[email protected]>>
Sent: Friday, June 30, 2017 10:55:39 PM
To: Christian König; Michel Dänzer
Cc: [email protected]<mailto:[email protected]>
Subject: RE: [PATCH] drm/amdgpu: Make KIQ read/write register routine be atomic

Hi , Christian
The new code actually will not use the fence function , it just need a memory that expose both CPU and GPU address . Do you really want to add the wrap functions that just expose the CPU and GPU address in this case ?

Regards
Shaoyun.liu

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: Friday, June 30, 2017 3:57 AM
To: Michel Dänzer; Liu, Shaoyun
Cc: [email protected]<mailto:[email protected]>
Subject: Re: [PATCH] drm/amdgpu: Make KIQ read/write register routine be atomic

Am 30.06.2017 um 03:21 schrieb Michel Dänzer:

Quoted 6 lines · Shaoyun Liu
Quoted 3 lines

1. Use spin lock instead of mutex in KIQ 2. Directly write to KIQ fence address instead of using fence_emit() 3. Disable the interrupt for KIQ read/write and use CPU polling

This list indicates that this patch should be split up in at least three patches. :)

Yeah, apart from that is is not a good idea to mess with the fence internals directly in the KIQ code, please add a helper in the fence code for this.

Regards,
Christian.
_______________________________________________
amd-gfx mailing list [email protected]<mailto:[email protected]>
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx