Skip to contentSkip to conversation

drm/amdgpu: fix a typo

PATCH In review Alex Xie 8 participants 37 messages Jun 22, 2017 – Jun 28, 2017 27 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

Alex Xie

author
#0 lore raw

PATCH 1/3 drm/amdgpu: fix a typo

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -1
index 7635f38..94c27fc 100644
@@ -494,7 +494,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,
 				 &e->user_invalidated) && e->user_pages) {
 
 				/* We acquired a page array, but somebody
-				 * invalidated it. Free it an try again
+				 * invalidated it. Free it and try again
 				 */
 				release_pages(e->user_pages,
 					      e->robj->tbo.ttm->num_pages,
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1498099356-31332-1-git-send-email-AlexBin.Xie-5C7GfCeVMHo@public.gmane.org

author
#2 lore raw

PATCH 2/3 drm/amdgpu: change a function to static function

The function is called only once inside the .c file.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h    | 1 -
 drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 +-
 2 files changed, 1 insertion(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +0 -1
index 6b7d2a1..7caf514 100644
@@ -1929,7 +1929,6 @@ void amdgpu_pci_config_reset(struct amdgpu_device *adev);
 bool amdgpu_need_post(struct amdgpu_device *adev);
 void amdgpu_update_display_priority(struct amdgpu_device *adev);
 
-int amdgpu_cs_parser_init(struct amdgpu_cs_parser *p, void *data);
 void amdgpu_cs_report_moved_bytes(struct amdgpu_device *adev, u64 num_bytes);
 void amdgpu_ttm_placement_from_domain(struct amdgpu_bo *abo, u32 domain);
 bool amdgpu_ttm_bo_is_amdgpu_bo(struct ttm_buffer_object *bo);
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -1
index 94c27fc..82131d7 100644
@@ -63,7 +63,7 @@ static int amdgpu_cs_user_fence_chunk(struct amdgpu_cs_parser *p,
 	return 0;
 }
 
-int amdgpu_cs_parser_init(struct amdgpu_cs_parser *p, void *data)
+static int amdgpu_cs_parser_init(struct amdgpu_cs_parser *p, void *data)
 {
 	struct amdgpu_fpriv *fpriv = p->filp->driver_priv;
 	struct amdgpu_vm *vm = &fpriv->vm;
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1498099356-31332-2-git-send-email-AlexBin.Xie-5C7GfCeVMHo@public.gmane.org

Reviewed-by
#4 lore raw

Re: [PATCH 2/3] drm/amdgpu: change a function to static function

Quoted 3 lines · Alex Xie

The function is called only once inside the .c file.

The shortlog should explicitly say "drm/amdgpu: Make amdgpu_cs_parser_init static". With that, this patch and patch 1 are

Signature · 6 lines
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
author
#5 lore raw

PATCH 3/3 drm/amdgpu: optimize out a spin lock Use atomic instead of spin lock.

In reply to a message not in the archive
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   5 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c     | 110 +++++++++++++++++++----------
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |   1 -
 3 files changed, 76 insertions(+), 40 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +2 -3
index 7caf514..21d318b 100644
@@ -1588,9 +1588,8 @@ struct amdgpu_device {
 
 	/* data for buffer migration throttling */
 	struct {
-		spinlock_t		lock;
-		s64			last_update_us;
-		s64			accum_us; /* accumulated microseconds */
+		atomic64_t		last_update_us;
+		atomic64_t		accum_us; /* accumulated microseconds */
 		u32			log2_max_MBps;
 	} mm_stats;
 
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +74 -36

136 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -1
index ff90f78..9e9d592 100644
@@ -2117,7 +2117,6 @@ int amdgpu_device_init(struct amdgpu_device *adev,
 	spin_lock_init(&adev->didt_idx_lock);
 	spin_lock_init(&adev->gc_cac_idx_lock);
 	spin_lock_init(&adev->audio_endpt_idx_lock);
-	spin_lock_init(&adev->mm_stats.lock);
 
 	INIT_LIST_HEAD(&adev->shadow_list);
 	mutex_init(&adev->shadow_list_lock);
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#6 lore raw
In reply to a message not in the archive

Am 22.06.2017 um 04:42 schrieb Alex Xie:

Quoted 1 line

With the commit message fixed as Michel suggested patches #1 and #2 are Reviewed-by: Christian König <[email protected]> as well.

On patch #3 Marek needs to take a look, cause I don't know the logic behind that.

Regards,
Christian.

Quoted 9 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0 -0
index 7635f38..94c27fc 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, · line 500

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -1
@@ -494,7 +494,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,
  				 &e->user_invalidated) && e->user_pages) {
 
  				/* We acquired a page array, but somebody
-				 * invalidated it. Free it an try again
+				 * invalidated it. Free it and try again
  				 */
  				release_pages(e->user_pages,
  					      e->robj->tbo.ttm->num_pages,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 5ff65f82-9d15-4606-7e25-e4f75c172aed-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

author Reviewed-by Christian König
#8 lore raw

Hi Christian,

In fact, the change from spinlock to atomic is quite painful. When I started, I thought it was easy but later I found there might be race condition here and there. Now I think the change looks more robust. In kernel source, there are several other drivers used the same trick.

On the other hand, I think the logic itself might be optimized considering the locking. But I had spent quite some effort to maintain original logic.

Thanks,

Alex Bin

From: Christian König <[email protected]>
Sent: Thursday, June 22, 2017 3:35 AM
To: Xie, AlexBin; [email protected]
Subject: Re: [PATCH 1/3] drm/amdgpu: fix a typo

Am 22.06.2017 um 04:42 schrieb Alex Xie:

Quoted 1 line

With the commit message fixed as Michel suggested patches #1 and #2 are

On patch #3 Marek needs to take a look, cause I don't know the logic behind that.

Regards,
Christian.

Quoted 9 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0 -0
index 7635f38..94c27fc 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, · line 500

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -1
@@ -494,7 +494,7 @@ static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p,
                                &e->user_invalidated) && e->user_pages) {
 
                               /* We acquired a page array, but somebody
-                              * invalidated it. Free it an try again
+                              * invalidated it. Free it and try again
                                */
                               release_pages(e->user_pages,
                                             e->robj->tbo.ttm->num_pages,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): DM5PR12MB1257B1FCC981D29A6D4045CDF2DB0-2J9CzHegvk/[email protected]

#10 lore raw

Hi Alex,

yeah, the change looks totally ok to me. The problem is just that I'm not familiar with that part of the source.

Marek came up with that, so he should at least take a look and nod.

Regards,
Christian.

Am 22.06.2017 um 17:33 schrieb Xie, AlexBin:

Quoted 54 lines

Hi Christian,

In fact, the change from spinlock to atomic is quite painful. When I started, I thought it was easy but later I found there might be race condition here and there. Now I think the change looks more robust. In kernel source, there are several other drivers used the same trick.

On the other hand, I think the logic itself might be optimized considering the locking. But I had spent quite some effort to maintain original logic.

Thanks,

Alex Bin

*From:* Christian König <[email protected]> *Sent:* Thursday, June 22, 2017 3:35 AM
*To:* Xie, AlexBin; [email protected] *Subject:* Re: [PATCH 1/3] drm/amdgpu: fix a typo
Am 22.06.2017 um 04:42 schrieb Alex Xie:

Quoted 1 line

With the commit message fixed as Michel suggested patches #1 and #2 are

On patch #3 Marek needs to take a look, cause I don't know the logic behind that.

Regards,
Christian.

Review

Quoted 5 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c

drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c static int amdgpu_cs_parser_bos(struct

Quoted 4 lines

index 7635f38..94c27fc 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +0 -0
@@ -494,7 +494,7 @@ static int amdgpu_cs_parser_bos(struct 

amdgpu_cs_parser *p,

drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c static int amdgpu_cs_parser_bos(struct amdgpu_cs_parser *p, · line 500

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +1 -1
 &e->user_invalidated) && e->user_pages) {
 
                               /* We acquired a page array, but somebody
-                              * invalidated it. Free it an try again
+                              * invalidated it. Free it and try again
                                */
 release_pages(e->user_pages,
 e->robj->tbo.ttm->num_pages,
Quoted 4 lines

_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#11 lore raw
In reply to a message not in the archive
Quoted 11 lines · AlexBin

Hi Christian,

In fact, the change from spinlock to atomic is quite painful. When I started, I thought it was easy but later I found there might be race condition here and there. Now I think the change looks more robust. In kernel source, there are several other drivers used the same trick.

On the other hand, I think the logic itself might be optimized considering the locking. But I had spent quite some effort to maintain original logic.

It seems quite complicated and I don't know if there is any performance benefit. Spinlocks are nice because they allow preemption.

It would be more interesting to merge the CS and BO_LIST ioctls into one.

Marek
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): CAAxE2A7ic2YLmnkzM0Faa3bO9-GwWRTHRc0EUVej5UU7Yhs4Qg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#13 lore raw

Hi Marek,

Spin lock and spin unlock is fast. But it is not so fast compared with atomic, which is a single CPU instruction in x86.

1. spinlock does NOT allow preemption at local CPU. Let us have a look at how spin lock was implemented.

static inline void __raw_spin_lock(raw_spinlock_t *lock)

{
    preempt_disable(); <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<--This is 
memory barrier operation too.
    spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
    LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
}

2. A function __lock_acquire called by spinlock. The function is so long that I would not attach all of it here.

There is atomic operation inside and 12 meta data updates and 14 if statements and it calls quite some other functions.

Note that it disable IRQ...

static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
int trylock, int read, int check, int hardirqs_off, struct lockdep_map *nest_lock, unsigned long ip, int references, int pin_count)

{
    struct task_struct *curr = current;
    struct lock_class *class = NULL;
    struct held_lock *hlock;
    unsigned int depth;
    int chain_head = 0;
    int class_idx;
    u64 chain_key;

    if (unlikely(!debug_locks))
        return 0;

    /*
     * Lockdep should run with IRQs disabled, otherwise we could
     * get an interrupt which would want to take locks, which would
     * end up in lockdep and have you got a head-ache already?
     */
    if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) <<<<<<<<<<<<<<<Disable IRQ
        return 0;

....

3. Another function called by spinlock in a higher level:

void lock_acquire(struct lockdep_map *lock, unsigned int subclass,

int trylock, int read, int check, struct lockdep_map *nest_lock, unsigned long ip)

{
    unsigned long flags;

    if (unlikely(current->lockdep_recursion))
        return;

    raw_local_irq_save(flags);
    check_flags(flags);

    current->lockdep_recursion = 1;
    trace_lock_acquire(lock, subclass, trylock, read, check, nest_lock, 
ip);
    __lock_acquire(lock, subclass, trylock, read, check,
               irqs_disabled_flags(flags), nest_lock, ip, 0, 0);
    current->lockdep_recursion = 0;
    raw_local_irq_restore(flags);
}

Thanks,

Alex Bin

Quoted 10 lines · Marek Olšák
Quoted 9 lines · AlexBin

Hi Christian,

In fact, the change from spinlock to atomic is quite painful. When I started, I thought it was easy but later I found there might be race condition here and there. Now I think the change looks more robust. In kernel source, there are several other drivers used the same trick.

Quoted 8 lines
Quoted 2 lines

On the other hand, I think the logic itself might be optimized considering the locking. But I had spent quite some effort to maintain original logic.

It seems quite complicated and I don't know if there is any performance benefit. Spinlocks are nice because they allow preemption.

It would be more interesting to merge the CS and BO_LIST ioctls into one.

Marek

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): efc2289a-382d-8443-8419-da2d0cadfd77-5C7GfCeVMHo@public.gmane.org

#15 lore raw

To clarify, local IRQ is disabled by calling raw_local_irq_save(flags);

Function __lock_acquire double checks that the local IRQ is really disabled.

Quoted 72 lines · axie

Hi Marek,

Spin lock and spin unlock is fast. But it is not so fast compared with atomic, which is a single CPU instruction in x86.

1. spinlock does NOT allow preemption at local CPU. Let us have a look at how spin lock was implemented.

static inline void __raw_spin_lock(raw_spinlock_t *lock)

{
    preempt_disable(); <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<--This is 
memory barrier operation too.
    spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
    LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
}

2. A function __lock_acquire called by spinlock. The function is so long that I would not attach all of it here.

There is atomic operation inside and 12 meta data updates and 14 if statements and it calls quite some other functions.

Note that it disable IRQ...

static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
int trylock, int read, int check, int hardirqs_off, struct lockdep_map *nest_lock, unsigned long ip, int references, int pin_count)

{
    struct task_struct *curr = current;
    struct lock_class *class = NULL;
    struct held_lock *hlock;
    unsigned int depth;
    int chain_head = 0;
    int class_idx;
    u64 chain_key;

    if (unlikely(!debug_locks))
        return 0;

    /*
     * Lockdep should run with IRQs disabled, otherwise we could
     * get an interrupt which would want to take locks, which would
     * end up in lockdep and have you got a head-ache already?
     */
    if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) <<<<<<<<<<<<<<<Disable IRQ
        return 0;

....

3. Another function called by spinlock in a higher level:

void lock_acquire(struct lockdep_map *lock, unsigned int subclass,

int trylock, int read, int check, struct lockdep_map *nest_lock, unsigned long ip)

{
    unsigned long flags;

    if (unlikely(current->lockdep_recursion))
        return;

    raw_local_irq_save(flags);
    check_flags(flags);

    current->lockdep_recursion = 1;
    trace_lock_acquire(lock, subclass, trylock, read, check, 
nest_lock, ip);
    __lock_acquire(lock, subclass, trylock, read, check,
               irqs_disabled_flags(flags), nest_lock, ip, 0, 0);
    current->lockdep_recursion = 0;
    raw_local_irq_restore(flags);
}
Quoted 25 lines

Thanks,

Alex Bin

Quoted 18 lines · Marek Olšák
Quoted 11 lines · AlexBin

Hi Christian,

In fact, the change from spinlock to atomic is quite painful. When I started, I thought it was easy but later I found there might be race condition here and there. Now I think the change looks more robust. In kernel source, there are several other drivers used the same trick.

On the other hand, I think the logic itself might be optimized considering the locking. But I had spent quite some effort to maintain original logic.

It seems quite complicated and I don't know if there is any performance benefit. Spinlocks are nice because they allow preemption.

It would be more interesting to merge the CS and BO_LIST ioctls into one.

Marek

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): e23f64c3-6abf-ff65-2300-5c44d242f4df-5C7GfCeVMHo@public.gmane.org

#17 lore raw

That's all nice, but does it improve performance? Have you been able to measure some performance difference with that code? Were you targeting a specific inefficiency you had seen e.g. with a CPU profiler?

Marek

Quoted 117 lines · axie

To clarify, local IRQ is disabled by calling raw_local_irq_save(flags);

Function __lock_acquire double checks that the local IRQ is really disabled.

Quoted 109 lines · axie

Hi Marek,

Spin lock and spin unlock is fast. But it is not so fast compared with atomic, which is a single CPU instruction in x86.

1. spinlock does NOT allow preemption at local CPU. Let us have a look at how spin lock was implemented.

static inline void __raw_spin_lock(raw_spinlock_t *lock)

{
    preempt_disable(); <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<--This is
memory barrier operation too.
    spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
    LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
}

2. A function __lock_acquire called by spinlock. The function is so long that I would not attach all of it here.

There is atomic operation inside and 12 meta data updates and 14 if statements and it calls quite some other functions.

Note that it disable IRQ...

static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
int trylock, int read, int check, int hardirqs_off, struct lockdep_map *nest_lock, unsigned long ip, int references, int pin_count)

{
    struct task_struct *curr = current;
    struct lock_class *class = NULL;
    struct held_lock *hlock;
    unsigned int depth;
    int chain_head = 0;
    int class_idx;
    u64 chain_key;

    if (unlikely(!debug_locks))
        return 0;

    /*
     * Lockdep should run with IRQs disabled, otherwise we could
     * get an interrupt which would want to take locks, which would
     * end up in lockdep and have you got a head-ache already?
     */
    if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) <<<<<<<<<<<<<<<Disable IRQ
        return 0;

....

3. Another function called by spinlock in a higher level:

void lock_acquire(struct lockdep_map *lock, unsigned int subclass,

int trylock, int read, int check, struct lockdep_map *nest_lock, unsigned long ip)

{
    unsigned long flags;

    if (unlikely(current->lockdep_recursion))
        return;

    raw_local_irq_save(flags);
    check_flags(flags);

    current->lockdep_recursion = 1;
    trace_lock_acquire(lock, subclass, trylock, read, check, nest_lock,
ip);
    __lock_acquire(lock, subclass, trylock, read, check,
               irqs_disabled_flags(flags), nest_lock, ip, 0, 0);
    current->lockdep_recursion = 0;
    raw_local_irq_restore(flags);
}

Thanks,

Alex Bin

Quoted 24 lines · Marek Olšák
Quoted 10 lines · AlexBin

Hi Christian,

In fact, the change from spinlock to atomic is quite painful. When I started, I thought it was easy but later I found there might be race condition here and there. Now I think the change looks more robust. In kernel source, there are several other drivers used the same trick.

Quoted 4 lines

On the other hand, I think the logic itself might be optimized considering
the locking. But I had spent quite some effort to maintain original logic.

It seems quite complicated and I don't know if there is any performance benefit. Spinlocks are nice because they allow preemption.

It would be more interesting to merge the CS and BO_LIST ioctls into one.

Marek

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): CAAxE2A61KqM9gr=Zoo5PHFNb8gWp74RG9KOs=efWzf1dMKBwSA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#19 lore raw

Hi Marek,

So do you agree that spinlock disables CPU preemption, contrary to your original idea?

If you have new reason that this patch does not improve, please speak out.

Many patches in GPU driver aim at improving performance and power efficiency. Does most patches submitted in AMDGPU requires a benchmarking first?

If all developers are required to always answer your questions when code review, I am afraid that most open source community developers cannot meet that requirement and stop working on AMDGPU.

To improve performance, there are many bottlenecks to clear. When the last several bottlenecks are clear, the performance will show faster more significantly.

My pass profiling experience told me that clearing a lock can improve performance for some driver like 0.3% to much bigger percentage. It depends on many factors, even depends on the application itself.

This is not the first bottleneck fixed. This is surely not the last one.

Thanks,

Alex Bin

Quoted 5 lines · Marek Olšák

That's all nice, but does it improve performance? Have you been able to measure some performance difference with that code? Were you targeting a specific inefficiency you had seen e.g. with a CPU profiler?

Quoted 114 lines

Marek

Quoted 111 lines · axie

To clarify, local IRQ is disabled by calling raw_local_irq_save(flags);

Function __lock_acquire double checks that the local IRQ is really disabled.

Quoted 104 lines · axie

Hi Marek,

Spin lock and spin unlock is fast. But it is not so fast compared with atomic, which is a single CPU instruction in x86.

1. spinlock does NOT allow preemption at local CPU. Let us have a look at how spin lock was implemented.

static inline void __raw_spin_lock(raw_spinlock_t *lock)

{
     preempt_disable(); <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<--This is
memory barrier operation too.
     spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
     LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
}

2. A function __lock_acquire called by spinlock. The function is so long that I would not attach all of it here.

There is atomic operation inside and 12 meta data updates and 14 if statements and it calls quite some other functions.

Note that it disable IRQ...

static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
int trylock, int read, int check, int hardirqs_off, struct lockdep_map *nest_lock, unsigned long ip, int references, int pin_count)

{
     struct task_struct *curr = current;
     struct lock_class *class = NULL;
     struct held_lock *hlock;
     unsigned int depth;
     int chain_head = 0;
     int class_idx;
     u64 chain_key;

     if (unlikely(!debug_locks))
         return 0;

     /*
      * Lockdep should run with IRQs disabled, otherwise we could
      * get an interrupt which would want to take locks, which would
      * end up in lockdep and have you got a head-ache already?
      */
     if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) <<<<<<<<<<<<<<<Disable IRQ
         return 0;

....

3. Another function called by spinlock in a higher level:

void lock_acquire(struct lockdep_map *lock, unsigned int subclass,

int trylock, int read, int check, struct lockdep_map *nest_lock, unsigned long ip)

{
     unsigned long flags;

     if (unlikely(current->lockdep_recursion))
         return;

     raw_local_irq_save(flags);
     check_flags(flags);

     current->lockdep_recursion = 1;
     trace_lock_acquire(lock, subclass, trylock, read, check, nest_lock,
ip);
     __lock_acquire(lock, subclass, trylock, read, check,
                irqs_disabled_flags(flags), nest_lock, ip, 0, 0);
     current->lockdep_recursion = 0;
     raw_local_irq_restore(flags);
}

Thanks,

Alex Bin

Quoted 21 lines · Marek Olšák
Quoted 9 lines · AlexBin

Hi Christian,

In fact, the change from spinlock to atomic is quite painful. When I started, I thought it was easy but later I found there might be race condition here and there. Now I think the change looks more robust. In kernel source, there are several other drivers used the same trick.

Quoted 4 lines

On the other hand, I think the logic itself might be optimized considering
the locking. But I had spent quite some effort to maintain original logic.

It seems quite complicated and I don't know if there is any performance benefit. Spinlocks are nice because they allow preemption.

It would be more interesting to merge the CS and BO_LIST ioctls into one.

Marek

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 7134d81f-a60f-7093-d2a3-70edde23cdb2-5C7GfCeVMHo@public.gmane.org

#21 lore raw

Hi Alex,

actually Marek is right, command submission is actually not much of a bottleneck to us because it is handled from a separate userspace thread.

So those micro optimizations you do here on CPU cycles are actually rather superfluous.

But giving the CS IOCTL an option for directly specifying the BOs instead of a BO list like Marek suggested would indeed save us some time here.

Regards,
Christian.

Am 23.06.2017 um 04:23 schrieb axie:

Quoted 147 lines

Hi Marek,

So do you agree that spinlock disables CPU preemption, contrary to your original idea?

If you have new reason that this patch does not improve, please speak out.

Many patches in GPU driver aim at improving performance and power efficiency. Does most patches submitted in AMDGPU requires a benchmarking first?

If all developers are required to always answer your questions when code review, I am afraid that most open source community developers cannot meet that requirement and stop working on AMDGPU.

To improve performance, there are many bottlenecks to clear. When the last several bottlenecks are clear, the performance will show faster more significantly.

My pass profiling experience told me that clearing a lock can improve performance for some driver like 0.3% to much bigger percentage. It depends on many factors, even depends on the application itself.

This is not the first bottleneck fixed. This is surely not the last one.

Thanks,

Alex Bin

Quoted 119 lines · Marek Olšák

That's all nice, but does it improve performance? Have you been able to measure some performance difference with that code? Were you targeting a specific inefficiency you had seen e.g. with a CPU profiler?

Marek

Quoted 111 lines · axie

To clarify, local IRQ is disabled by calling raw_local_irq_save(flags);

Function __lock_acquire double checks that the local IRQ is really disabled.

Quoted 104 lines · axie

Hi Marek,

Spin lock and spin unlock is fast. But it is not so fast compared with atomic, which is a single CPU instruction in x86.

1. spinlock does NOT allow preemption at local CPU. Let us have a look at how spin lock was implemented.

static inline void __raw_spin_lock(raw_spinlock_t *lock)

{
     preempt_disable(); <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<--This is
memory barrier operation too.
     spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
     LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
}

2. A function __lock_acquire called by spinlock. The function is so long that I would not attach all of it here.

There is atomic operation inside and 12 meta data updates and 14 if statements and it calls quite some other functions.

Note that it disable IRQ...

static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
int trylock, int read, int check, int hardirqs_off, struct lockdep_map *nest_lock, unsigned long ip, int references, int pin_count)

{
     struct task_struct *curr = current;
     struct lock_class *class = NULL;
     struct held_lock *hlock;
     unsigned int depth;
     int chain_head = 0;
     int class_idx;
     u64 chain_key;

     if (unlikely(!debug_locks))
         return 0;

     /*
      * Lockdep should run with IRQs disabled, otherwise we could
      * get an interrupt which would want to take locks, which would
      * end up in lockdep and have you got a head-ache already?
      */
     if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) 
<<<<<<<<<<<<<<<Disable IRQ
         return 0;

....

3. Another function called by spinlock in a higher level:

void lock_acquire(struct lockdep_map *lock, unsigned int subclass,

int trylock, int read, int check, struct lockdep_map *nest_lock, unsigned long ip)

{
     unsigned long flags;

     if (unlikely(current->lockdep_recursion))
         return;

     raw_local_irq_save(flags);
     check_flags(flags);

     current->lockdep_recursion = 1;
     trace_lock_acquire(lock, subclass, trylock, read, check, 
nest_lock,
ip);
     __lock_acquire(lock, subclass, trylock, read, check,
                irqs_disabled_flags(flags), nest_lock, ip, 0, 0);
     current->lockdep_recursion = 0;
     raw_local_irq_restore(flags);
}

Thanks,

Alex Bin

Quoted 20 lines · Marek Olšák
Quoted 9 lines · AlexBin

Hi Christian,

In fact, the change from spinlock to atomic is quite painful. When I started, I thought it was easy but later I found there might be race condition here and there. Now I think the change looks more robust. In kernel source, there are several other drivers used the same trick.

Quoted 4 lines

On the other hand, I think the logic itself might be optimized considering
the locking. But I had spent quite some effort to maintain original logic.

It seems quite complicated and I don't know if there is any performance benefit. Spinlocks are nice because they allow preemption.

It would be more interesting to merge the CS and BO_LIST ioctls into one.

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

Marek

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 3436ae97-39b1-4a3f-bb73-2991adad5715-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

#23 lore raw
Quoted 1 line · Christian König

But giving the CS IOCTL an option for directly specifying the BOs instead of a BO list like Marek suggested would indeed save us some time here.

interesting, I always follow how to improve our cs ioctl, since UMD guys aften complain our command submission is slower than windows. Then how to directly specifying the BOs instead of a BO list? BO handle array from UMD? Could your guys describe more clear? Is it doable?

Regards,
David Zhou
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): [email protected]

#25 lore raw

Am 23.06.2017 um 09:09 schrieb zhoucm1:

Quoted 6 lines · Christian König
Quoted 1 line

But giving the CS IOCTL an option for directly specifying the BOs instead of a BO list like Marek suggested would indeed save us some time here.

interesting, I always follow how to improve our cs ioctl, since UMD guys aften complain our command submission is slower than windows. Then how to directly specifying the BOs instead of a BO list? BO handle array from UMD? Could your guys describe more clear? Is it doable?

Making the BO list part of the CS IOCTL wouldn't help at all for the close source UMDs. To be precise we actually came up with the BO list approach because of their requirement.

The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status. It doesn't matter if the BOs come from the BO list or directly in the CS IOCTL.

The key point is that CS overhead is pretty much irrelevant for the open source stack, since Mesa does command submission from a separate thread anyway.

Regards,
Christian.

Quoted 3 lines

Regards,
David Zhou

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 0d847baf-8296-f3ce-7a8d-8823f33e392e-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

#27 lore raw
Quoted 11 lines · Christian König

Am 23.06.2017 um 09:09 schrieb zhoucm1:

Quoted 6 lines · Christian König
Quoted 1 line

But giving the CS IOCTL an option for directly specifying the BOs instead of a BO list like Marek suggested would indeed save us some time here.

interesting, I always follow how to improve our cs ioctl, since UMD guys aften complain our command submission is slower than windows. Then how to directly specifying the BOs instead of a BO list? BO handle array from UMD? Could your guys describe more clear? Is it doable?

Making the BO list part of the CS IOCTL wouldn't help at all for the close source UMDs. To be precise we actually came up with the BO list approach because of their requirement.

The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status.

Totally agree. Every time when I read code there, I often want to optimize them.

Quoted 3 lines

It doesn't matter if the BOs come from the BO list or directly in the CS IOCTL.

The key point is that CS overhead is pretty much irrelevant for the open source stack, since Mesa does command submission from a separate thread anyway.

If irrelevant for the open stack, then how does open source stack handle "The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status."? If open stack has a better way, I think closed stack can follow it, I don't know the history.

Regards,
David Zhou

Quoted 9 lines

Regards,
Christian.

Quoted 3 lines

Regards,
David Zhou

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#28 lore raw
In reply to a message not in the archive
Quoted 11 lines · Christian König

Am 23.06.2017 um 09:09 schrieb zhoucm1:

Quoted 6 lines · Christian König
Quoted 1 line

But giving the CS IOCTL an option for directly specifying the BOs instead of a BO list like Marek suggested would indeed save us some time here.

interesting, I always follow how to improve our cs ioctl, since UMD guys aften complain our command submission is slower than windows. Then how to directly specifying the BOs instead of a BO list? BO handle array from UMD? Could your guys describe more clear? Is it doable?

Making the BO list part of the CS IOCTL wouldn't help at all for the close source UMDs. To be precise we actually came up with the BO list approach because of their requirement.

The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status.

Totally agree. Every time when I read code there, I often want to optimize them.

Quoted 3 lines

It doesn't matter if the BOs come from the BO list or directly in the CS IOCTL.

The key point is that CS overhead is pretty much irrelevant for the open source stack, since Mesa does command submission from a separate thread anyway.

If irrelevant for the open stack, then how does open source stack handle "The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status."? If open stack has a better way, I think closed stack can follow it, I don't know the history.

Regards,
David Zhou

Quoted 9 lines

Regards,
Christian.

Quoted 3 lines

Regards,
David Zhou

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): [email protected]

#30 lore raw
Quoted 13 lines · On 2017年06月23日 17:01, zhoucm1 wrote:
Quoted 10 lines · Christian König

Am 23.06.2017 um 09:09 schrieb zhoucm1:

Quoted 6 lines · Christian König
Quoted 1 line

But giving the CS IOCTL an option for directly specifying the BOs instead of a BO list like Marek suggested would indeed save us some time here.

interesting, I always follow how to improve our cs ioctl, since UMD guys aften complain our command submission is slower than windows. Then how to directly specifying the BOs instead of a BO list? BO handle array from UMD? Could your guys describe more clear? Is it doable?

Making the BO list part of the CS IOCTL wouldn't help at all for the close source UMDs. To be precise we actually came up with the BO list approach because of their requirement.

Quoted 8 lines
Quoted 1 line

The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status.

Totally agree. Every time when I read code there, I often want to optimize them.

Quoted 3 lines

It doesn't matter if the BOs come from the BO list or directly in the CS IOCTL.

The key point is that CS overhead is pretty much irrelevant for the open source stack, since Mesa does command submission from a separate thread anyway.

If irrelevant for the open stack, then how does open source stack handle "The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status."? If open stack has a better way, I think closed stack can follow it, I don't know the history.

Do you not use bo list at all in mesa? radv as well?

Regards,
David Zhou

Quoted 13 lines

Regards,
David Zhou

Quoted 9 lines

Regards,
Christian.

Quoted 3 lines

Regards,
David Zhou

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): [email protected]

#32 lore raw

Am 23.06.2017 um 11:08 schrieb zhoucm1:

Quoted 16 lines · On 2017年06月23日 17:01, zhoucm1 wrote:
Quoted 13 lines · Christian König
Quoted 10 lines

Am 23.06.2017 um 09:09 schrieb zhoucm1:

Quoted 6 lines · Christian König
Quoted 1 line

But giving the CS IOCTL an option for directly specifying the BOs instead of a BO list like Marek suggested would indeed save us some time here.

interesting, I always follow how to improve our cs ioctl, since UMD guys aften complain our command submission is slower than windows. Then how to directly specifying the BOs instead of a BO list? BO handle array from UMD? Could your guys describe more clear? Is it doable?

Making the BO list part of the CS IOCTL wouldn't help at all for the close source UMDs. To be precise we actually came up with the BO list approach because of their requirement.

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status.

Totally agree. Every time when I read code there, I often want to optimize them.

Quoted 3 lines

It doesn't matter if the BOs come from the BO list or directly in the CS IOCTL.

The key point is that CS overhead is pretty much irrelevant for the open source stack, since Mesa does command submission from a separate thread anyway.

If irrelevant for the open stack, then how does open source stack handle "The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status."?

Command submission on the open stack is outsourced to a separate user space thread. E.g. when an application triggers a flush the IBs created so far are just put on a queue and another thread pushes them down to the kernel.

I mean reducing the overhead of the CS IOCTL is always nice, but you usual won't see any fps increase as long as not all CPUs are completely bound to some tasks.

Quoted 2 lines
Quoted 1 line

If open stack has a better way, I think closed stack can follow it, I don't know the history.

Do you not use bo list at all in mesa? radv as well?

I don't think so. Mesa just wants to send the list of used BOs down to the kernel with every IOCTL.

Regards,
Christian.

Quoted 21 lines

Regards,
David Zhou

Quoted 17 lines

Regards,
David Zhou

Quoted 9 lines

Regards,
Christian.

Quoted 3 lines

Regards,
David Zhou

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 807f27d6-7e53-4066-c440-699bf66dd227-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

#34 lore raw
Quoted 56 lines · Christian König

Am 23.06.2017 um 11:08 schrieb zhoucm1:

Quoted 38 lines · On 2017年06月23日 17:01, zhoucm1 wrote:
Quoted 34 lines · Christian König
Quoted 17 lines

Am 23.06.2017 um 09:09 schrieb zhoucm1:

Quoted 13 lines · Christian König
Quoted 4 lines

But giving the CS IOCTL an option for directly specifying the BOs instead of a BO list like Marek suggested would indeed save us some time here.

interesting, I always follow how to improve our cs ioctl, since UMD guys aften complain our command submission is slower than windows. Then how to directly specifying the BOs instead of a BO list? BO handle array from UMD? Could your guys describe more clear? Is it doable?

Quoted 6 lines

Making the BO list part of the CS IOCTL wouldn't help at all for the close source UMDs. To be precise we actually came up with the BO list approach because of their requirement.

The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status.

Totally agree. Every time when I read code there, I often want to optimize them.

Quoted 3 lines

It doesn't matter if the BOs come from the BO list or directly in the CS IOCTL.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

The key point is that CS overhead is pretty much irrelevant for the open source stack, since Mesa does command submission from a separate thread anyway.

If irrelevant for the open stack, then how does open source stack handle "The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status."?

Command submission on the open stack is outsourced to a separate user space thread. E.g. when an application triggers a flush the IBs created so far are just put on a queue and another thread pushes them down to the kernel.

I mean reducing the overhead of the CS IOCTL is always nice, but you usual won't see any fps increase as long as not all CPUs are completely bound to some tasks.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

If open stack has a better way, I think closed stack can follow it, I don't know the history.

Do you not use bo list at all in mesa? radv as well?

I don't think so. Mesa just wants to send the list of used BOs down to the kernel with every IOCTL.

The CS ioctl actually costs us some performance, but not as much as on closed source drivers.

MesaGL always executes all CS ioctls in a separate thread (in parallel with the UMD) except for the last IB that's submitted by SwapBuffers. SwapBuffers requires that all IBs have been submitted when SwapBuffers returns. For example, if you have 5 IBs per frame, 4 of them are executed on the thread and the overhead is hidden. The last one is executed on the thread too, but this time the Mesa driver has to wait for it. For things like glxgears with only 1 IB per frame, the thread doesn't hide anything and Mesa always has to wait for it after submission, just because of SwapBuffers.

Having 10 or more IBs per frame is great, because 9 are done in parallel and the last one is synchronous. The final CPU cost is 10x lower, but it's not zero.

For us, it's certainly useful to optimize the CS ioctl because of apps that submit only 1 IB per frame where multithreading has no effect or may even hurt performance.

The most obvious inefficiency is the BO_LIST ioctl that is completely unnecessary and only slows us down. What we need is exactly what radeon does.

Marek
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): CAAxE2A4vdV+QD55RAn+mrD92o39X_vPT6wLKCwkREx2=fpU2Vw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#36 lore raw

________________________________________
From: Marek Olšák [[email protected]]
Sent: Friday, June 23, 2017 6:49 PM
To: Christian König
Cc: Zhou, David(ChunMing); Xie, AlexBin; [email protected]; Xie, AlexBin
Subject: Re: [PATCH 1/3] drm/amdgpu: fix a typo

Quoted 56 lines · Christian König

Am 23.06.2017 um 11:08 schrieb zhoucm1:

Quoted 38 lines · On 2017年06月23日 17:01, zhoucm1 wrote:
Quoted 34 lines · Christian König
Quoted 17 lines

Am 23.06.2017 um 09:09 schrieb zhoucm1:

Quoted 13 lines · Christian König
Quoted 4 lines

But giving the CS IOCTL an option for directly specifying the BOs instead of a BO list like Marek suggested would indeed save us some time here.

interesting, I always follow how to improve our cs ioctl, since UMD guys aften complain our command submission is slower than windows. Then how to directly specifying the BOs instead of a BO list? BO handle array from UMD? Could your guys describe more clear? Is it doable?

Quoted 6 lines

Making the BO list part of the CS IOCTL wouldn't help at all for the close source UMDs. To be precise we actually came up with the BO list approach because of their requirement.

The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status.

Totally agree. Every time when I read code there, I often want to optimize them.

Quoted 3 lines

It doesn't matter if the BOs come from the BO list or directly in the CS IOCTL.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

The key point is that CS overhead is pretty much irrelevant for the open source stack, since Mesa does command submission from a separate thread anyway.

If irrelevant for the open stack, then how does open source stack handle "The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status."?

Command submission on the open stack is outsourced to a separate user space thread. E.g. when an application triggers a flush the IBs created so far are just put on a queue and another thread pushes them down to the kernel.

I mean reducing the overhead of the CS IOCTL is always nice, but you usual won't see any fps increase as long as not all CPUs are completely bound to some tasks.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

If open stack has a better way, I think closed stack can follow it, I don't know the history.

Do you not use bo list at all in mesa? radv as well?

I don't think so. Mesa just wants to send the list of used BOs down to the kernel with every IOCTL.

The CS ioctl actually costs us some performance, but not as much as on closed source drivers.

MesaGL always executes all CS ioctls in a separate thread (in parallel with the UMD) except for the last IB that's submitted by SwapBuffers. SwapBuffers requires that all IBs have been submitted when SwapBuffers returns. For example, if you have 5 IBs per frame, 4 of them are executed on the thread and the overhead is hidden. The last one is executed on the thread too, but this time the Mesa driver has to wait for it. For things like glxgears with only 1 IB per frame, the thread doesn't hide anything and Mesa always has to wait for it after submission, just because of SwapBuffers.

Having 10 or more IBs per frame is great, because 9 are done in
parallel and the last one is synchronous. The final CPU cost is 10x
lower, but it's not zero.
[DZ] Thanks Marek, this is very useful and helpful message for me to understand Mesa action of CS, I will talk to closed guys to see if it can be used for them. Anothing I also want to confirm with you, do you know if radv is using this cs way?

For us, it's certainly useful to optimize the CS ioctl because of apps that submit only 1 IB per frame where multithreading has no effect or may even hurt performance.

The most obvious inefficiency is the BO_LIST ioctl that is completely unnecessary and only slows us down. What we need is exactly what radeon does.

[DZ] I don't know how radeon handle bo list, could you describe it as well?

Thanks,
David Zhou

Marek
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): MWHPR1201MB02063CAC83DEEAB81EFA9A10B4D80-3iK1xFAIwjrUF/YbdlDdgWrFom/aUZj6nBOFsp37pqbUKgpGm//BTAC/[email protected]

#38 lore raw
Quoted 98 lines · David(ChunMing)

________________________________________
From: Marek Olšák [[email protected]]
Sent: Friday, June 23, 2017 6:49 PM
To: Christian König
Cc: Zhou, David(ChunMing); Xie, AlexBin; [email protected]; Xie, AlexBin
Subject: Re: [PATCH 1/3] drm/amdgpu: fix a typo

Quoted 56 lines · Christian König

Am 23.06.2017 um 11:08 schrieb zhoucm1:

Quoted 38 lines · On 2017年06月23日 17:01, zhoucm1 wrote:
Quoted 34 lines · Christian König
Quoted 17 lines

Am 23.06.2017 um 09:09 schrieb zhoucm1:

Quoted 13 lines · Christian König
Quoted 4 lines

But giving the CS IOCTL an option for directly specifying the BOs instead of a BO list like Marek suggested would indeed save us some time here.

interesting, I always follow how to improve our cs ioctl, since UMD guys aften complain our command submission is slower than windows. Then how to directly specifying the BOs instead of a BO list? BO handle array from UMD? Could your guys describe more clear? Is it doable?

Quoted 6 lines

Making the BO list part of the CS IOCTL wouldn't help at all for the close source UMDs. To be precise we actually came up with the BO list approach because of their requirement.

The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status.

Totally agree. Every time when I read code there, I often want to optimize them.

Quoted 3 lines

It doesn't matter if the BOs come from the BO list or directly in the CS IOCTL.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

The key point is that CS overhead is pretty much irrelevant for the open source stack, since Mesa does command submission from a separate thread anyway.

If irrelevant for the open stack, then how does open source stack handle "The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status."?

Command submission on the open stack is outsourced to a separate user space thread. E.g. when an application triggers a flush the IBs created so far are just put on a queue and another thread pushes them down to the kernel.

I mean reducing the overhead of the CS IOCTL is always nice, but you usual won't see any fps increase as long as not all CPUs are completely bound to some tasks.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

If open stack has a better way, I think closed stack can follow it, I don't know the history.

Do you not use bo list at all in mesa? radv as well?

I don't think so. Mesa just wants to send the list of used BOs down to the kernel with every IOCTL.

The CS ioctl actually costs us some performance, but not as much as on closed source drivers.

MesaGL always executes all CS ioctls in a separate thread (in parallel with the UMD) except for the last IB that's submitted by SwapBuffers. SwapBuffers requires that all IBs have been submitted when SwapBuffers returns. For example, if you have 5 IBs per frame, 4 of them are executed on the thread and the overhead is hidden. The last one is executed on the thread too, but this time the Mesa driver has to wait for it. For things like glxgears with only 1 IB per frame, the thread doesn't hide anything and Mesa always has to wait for it after submission, just because of SwapBuffers.

Having 10 or more IBs per frame is great, because 9 are done in
parallel and the last one is synchronous. The final CPU cost is 10x
lower, but it's not zero.
[DZ] Thanks Marek, this is very useful and helpful message for me to understand Mesa action of CS, I will talk to closed guys to see if it can be used for them. Anothing I also want to confirm with you, do you know if radv is using this cs way?

For us, it's certainly useful to optimize the CS ioctl because of apps that submit only 1 IB per frame where multithreading has no effect or may even hurt performance.

Quoted 5 lines

The most obvious inefficiency is the BO_LIST ioctl that is completely unnecessary and only slows us down. What we need is exactly what radeon does.

[DZ] I don't know how radeon handle bo list, could you describe it as well?

Inputs for the following ioctls are:

AMDGPU: BO_LIST:
- list of BOs

AMDGPU: CS
- list of IBs
- BO list handle

RADEON: CS
- one IB
- list of BOs

Ideal solution for a new amdgpu CS ioctl:
- list of IBs
- list of BOs

Marek
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): CAAxE2A7jmhxC8SctSSxXhc7XRwSgzmPOOF5HmmtAPL7aYKE+Jg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#40 lore raw
Quoted 115 lines · Marek Olšák
Quoted 98 lines · David(ChunMing)

________________________________________
From: Marek Olšák [[email protected]]
Sent: Friday, June 23, 2017 6:49 PM
To: Christian König
Cc: Zhou, David(ChunMing); Xie, AlexBin; [email protected]; Xie, AlexBin
Subject: Re: [PATCH 1/3] drm/amdgpu: fix a typo

Quoted 56 lines · Christian König

Am 23.06.2017 um 11:08 schrieb zhoucm1:

Quoted 38 lines · On 2017年06月23日 17:01, zhoucm1 wrote:
Quoted 34 lines · Christian König
Quoted 17 lines

Am 23.06.2017 um 09:09 schrieb zhoucm1:

Quoted 13 lines · Christian König
Quoted 4 lines

But giving the CS IOCTL an option for directly specifying the BOs instead of a BO list like Marek suggested would indeed save us some time here.

interesting, I always follow how to improve our cs ioctl, since UMD guys aften complain our command submission is slower than windows. Then how to directly specifying the BOs instead of a BO list? BO handle array from UMD? Could your guys describe more clear? Is it doable?

Quoted 6 lines

Making the BO list part of the CS IOCTL wouldn't help at all for the close source UMDs. To be precise we actually came up with the BO list approach because of their requirement.

The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status.

Totally agree. Every time when I read code there, I often want to optimize them.

Quoted 3 lines

It doesn't matter if the BOs come from the BO list or directly in the CS IOCTL.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

The key point is that CS overhead is pretty much irrelevant for the open source stack, since Mesa does command submission from a separate thread anyway.

If irrelevant for the open stack, then how does open source stack handle "The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status."?

Command submission on the open stack is outsourced to a separate user space thread. E.g. when an application triggers a flush the IBs created so far are just put on a queue and another thread pushes them down to the kernel.

I mean reducing the overhead of the CS IOCTL is always nice, but you usual won't see any fps increase as long as not all CPUs are completely bound to some tasks.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

If open stack has a better way, I think closed stack can follow it, I don't know the history.

Do you not use bo list at all in mesa? radv as well?

I don't think so. Mesa just wants to send the list of used BOs down to the kernel with every IOCTL.

The CS ioctl actually costs us some performance, but not as much as on closed source drivers.

MesaGL always executes all CS ioctls in a separate thread (in parallel with the UMD) except for the last IB that's submitted by SwapBuffers. SwapBuffers requires that all IBs have been submitted when SwapBuffers returns. For example, if you have 5 IBs per frame, 4 of them are executed on the thread and the overhead is hidden. The last one is executed on the thread too, but this time the Mesa driver has to wait for it. For things like glxgears with only 1 IB per frame, the thread doesn't hide anything and Mesa always has to wait for it after submission, just because of SwapBuffers.

Having 10 or more IBs per frame is great, because 9 are done in
parallel and the last one is synchronous. The final CPU cost is 10x
lower, but it's not zero.
[DZ] Thanks Marek, this is very useful and helpful message for me to understand Mesa action of CS, I will talk to closed guys to see if it can be used for them. Anothing I also want to confirm with you, do you know if radv is using this cs way?

For us, it's certainly useful to optimize the CS ioctl because of apps that submit only 1 IB per frame where multithreading has no effect or may even hurt performance.

Quoted 5 lines

The most obvious inefficiency is the BO_LIST ioctl that is completely unnecessary and only slows us down. What we need is exactly what radeon does.

[DZ] I don't know how radeon handle bo list, could you describe it as well?

Inputs for the following ioctls are:

AMDGPU: BO_LIST:
- list of BOs

AMDGPU: CS
- list of IBs
- BO list handle

Quoted 7 lines

RADEON: CS
- one IB
- list of BOs

Ideal solution for a new amdgpu CS ioctl:
- list of IBs
- list of BOs

I'd like to say that the current CS ioctl design is only a half of the problem with slow command submission. The second half is the libdrm overhead itself. Having wrapper functions around ioctls that have to unwrap input objects into alloca'd memory just for the ioctl to be called is simply wasted CPU time. There are cases where libdrm is useful. Command submission is not one of them. Any driver developer or vendor putting CS wrappers into libdrm is putting himself into a losing position. It's a perpetuated community myth that libdrm should have wrappers for everything.

Our winsys in Mesa is designed such that it can call the CS ioctl right when it's requested. The whole CS ioctl input structure is always ready at any point in time, because it's updated incrementally while draw calls are made. The radeon winsys works that way. The amdgpu winsys works that way too, but there is another translation of inputs in libdrm that defeats it. Intel also publicly admitted that putting CS wrappers into libdrm was stupid.

The solution for the best performance is to call the CS ioctl directly from UMDs, i.e. Mesa and Vulkan should call drmCommandWriteRead(fd, DRM_AMDGPU_CS2, ...) directly.

Until that's done, the command submission for the radeon kernel driver will remain faster than amdgpu.

Marek
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#41 lore raw
In reply to a message not in the archive
Quoted 54 lines · Marek Olšák
Quoted 52 lines · Christian König

Am 23.06.2017 um 11:08 schrieb zhoucm1:

Quoted 28 lines · On 2017年06月23日 17:01, zhoucm1 wrote:
Quoted 27 lines · Christian König
Quoted 13 lines

Am 23.06.2017 um 09:09 schrieb zhoucm1:

Quoted 10 lines · Christian König
Quoted 4 lines

But giving the CS IOCTL an option for directly specifying the BOs instead of a BO list like Marek suggested would indeed save us some time here.

interesting, I always follow how to improve our cs ioctl, since UMD guys aften complain our command submission is slower than windows. Then how to directly specifying the BOs instead of a BO list? BO handle array from UMD? Could your guys describe more clear? Is it doable?

Quoted 6 lines

Making the BO list part of the CS IOCTL wouldn't help at all for the close source UMDs. To be precise we actually came up with the BO list approach because of their requirement.

The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status.

Totally agree. Every time when I read code there, I often want to optimize them.

Quoted 3 lines

It doesn't matter if the BOs come from the BO list or directly in the CS IOCTL.

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

The key point is that CS overhead is pretty much irrelevant for the open source stack, since Mesa does command submission from a separate thread anyway.

If irrelevant for the open stack, then how does open source stack handle "The biggest bunch of work during CS is reserving all the buffers, validating them and checking their VM status."?

Command submission on the open stack is outsourced to a separate user space thread. E.g. when an application triggers a flush the IBs created so far are just put on a queue and another thread pushes them down to the kernel.

I mean reducing the overhead of the CS IOCTL is always nice, but you usual won't see any fps increase as long as not all CPUs are completely bound to some tasks.

Quoted 4 lines
Quoted 2 lines

If open stack has a better way, I think closed stack can follow it, I don't know the history.

Do you not use bo list at all in mesa? radv as well?

Quoted 8 lines
Quoted 2 lines

I don't think so. Mesa just wants to send the list of used BOs down to the kernel with every IOCTL.

The CS ioctl actually costs us some performance, but not as much as on closed source drivers.

MesaGL always executes all CS ioctls in a separate thread (in parallel with the UMD) except for the last IB that's submitted by SwapBuffers.

... or by an explicit glFinish or glFlush (at least when the current draw buffer isn't a back buffer) call, right?

Quoted 3 lines

For us, it's certainly useful to optimize the CS ioctl because of apps that submit only 1 IB per frame where multithreading has no effect or may even hurt performance.

Another possibility might be flushing earlier, e.g. when the GPU and/or CS submission thread are idle. But optimizing the CS ioctl would still help in that case.

Finding good heuristics which allows better utilization of the GPU / CS submission thread and doesn't hurt performance in any scenario might be tricky though.

Signature · 6 lines
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#42 lore raw
In reply to a message not in the archive
Quoted 1 line

Do you not use bo list at all in mesa? radv as well?

Currently radv is creating a bo list per command submission. radv does not use an offload thread to do command submission, as it seems pretty un-vulkan to use a thread for the queue submission thread the game uses.

I have considered investigating this, but with vulkan it's probably optimising for the single threaded case which isn't where apps should really be.

At the moment we regenerate the bo list on every CS ioctl, we probably can do a lot better, I know Bas has looked into this area a bit more than I.

Dave.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): CAPM=9tzXrih9qpkOY+OGC1TF8nD0ZmwNnxXiRbMtcO7tR1-m1Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#44 lore raw
Quoted 6 lines · Dave Airlie
Quoted 1 line

Do you not use bo list at all in mesa? radv as well?

Currently radv is creating a bo list per command submission. radv does not use an offload thread to do command submission, as it seems pretty un-vulkan to use a thread for the queue submission thread the game uses.

Quoted 7 lines

I have considered investigating this, but with vulkan it's probably optimising for the single threaded case which isn't where apps should really be.

At the moment we regenerate the bo list on every CS ioctl, we probably can do a lot better, I know Bas has looked into this area a bit more than I.

Thanks Dave for inputting.

Could I ask more about radv? How does radv make bo list for every cs ioctl? Adding filter in every operation, any related bo will be add to bo list during make command submission?

Thanks,
David Zhou

Quoted 2 lines

Dave.

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): [email protected]

#46 lore raw
Quoted 19 lines · On 26 June 2017 at 19:10, zhoucm1 <[email protected]> wrote:
Quoted 8 lines · Dave Airlie
Quoted 2 lines

Do you not use bo list at all in mesa? radv as well?

Currently radv is creating a bo list per command submission. radv does not use an offload thread to do command submission, as it seems pretty un-vulkan to use a thread for the queue submission thread the game uses.

Quoted 7 lines

I have considered investigating this, but with vulkan it's probably optimising for the single threaded case which isn't where apps should really be.

At the moment we regenerate the bo list on every CS ioctl, we probably can do a lot better, I know Bas has looked into this area a bit more than I.

Quoted 5 lines

Thanks Dave for inputting.

Could I ask more about radv? How does radv make bo list for every cs ioctl? Adding filter in every operation, any related bo will be add to bo list during make command submission?

When we create command buffers we add each bo to a list for it, but we don't call the kernel ioctl.

When we get a QueueSubmit, we usually submit two IBs to the kernel, one with some cache flushes and one with the main IB in it, however as we have to give the kernel one bo list we have to combine the two lists (even if the cache flush list only has one ib in it, itself), so we can't call create bo list early, we have to do it just before command submission. However I don't think we've had any problems with this being our slow path yet, but this may change in the future.

Dave.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): CAPM=9tw-E39oQ1gRPJs3oZBniALzgoa53ErF-trgxiN8Z9xyww-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#48 lore raw
Quoted 29 lines · Dave Airlie
Quoted 19 lines · On 26 June 2017 at 19:10, zhoucm1 <[email protected]> wrote:
Quoted 8 lines · Dave Airlie
Quoted 2 lines

Do you not use bo list at all in mesa? radv as well?

Currently radv is creating a bo list per command submission. radv does not use an offload thread to do command submission, as it seems pretty un-vulkan to use a thread for the queue submission thread the game uses.

Quoted 7 lines

I have considered investigating this, but with vulkan it's probably optimising for the single threaded case which isn't where apps should really be.

At the moment we regenerate the bo list on every CS ioctl, we probably can do a lot better, I know Bas has looked into this area a bit more than I.

Quoted 5 lines

Thanks Dave for inputting.

Could I ask more about radv? How does radv make bo list for every cs ioctl? Adding filter in every operation, any related bo will be add to bo list during make command submission?

When we create command buffers we add each bo to a list for it, but we don't call the kernel ioctl.

Quoted 7 lines

When we get a QueueSubmit, we usually submit two IBs to the kernel, one with some cache flushes and one with the main IB in it, however as we have to give the kernel one bo list we have to combine the two lists (even if the cache flush list only has one ib in it, itself), so we can't call create bo list early, we have to do it just before command submission. However I don't think we've had any problems with this being our slow path yet, but this may change in the future.

Yeah. IIRC the bo list create is typically < 10% of our kernel time for radv.

Quoted 2 lines

Dave.

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): CAP+8YyGouX0AeBxj+UNmb1Q7iwAzt0Z_hx__hf1Bb=YqC55J_A-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#50 lore raw
Quoted 25 lines · Bas Nieuwenhuizen
Quoted 24 lines · Dave Airlie
Quoted 12 lines · On 26 June 2017 at 19:10, zhoucm1 <[email protected]> wrote:
Quoted 10 lines · Dave Airlie
Quoted 1 line

Do you not use bo list at all in mesa? radv as well?

Currently radv is creating a bo list per command submission. radv does not use an offload thread to do command submission, as it seems pretty un-vulkan to use a thread for the queue submission thread the game uses.

I have considered investigating this, but with vulkan it's probably optimising for the single threaded case which isn't where apps should really be.

Quoted 8 lines
Quoted 3 lines

At the moment we regenerate the bo list on every CS ioctl, we probably can do a lot better, I know Bas has looked into this area a bit more than I.

Thanks Dave for inputting.

Could I ask more about radv? How does radv make bo list for every cs ioctl? Adding filter in every operation, any related bo will be add to bo list during make command submission?

When we create command buffers we add each bo to a list for it, but we don't call the kernel ioctl.

Quoted 8 lines
Quoted 7 lines

When we get a QueueSubmit, we usually submit two IBs to the kernel, one with some cache flushes and one with the main IB in it, however as we have to give the kernel one bo list we have to combine the two lists (even if the cache flush list only has one ib in it, itself), so we can't call create bo list early, we have to do it just before command submission. However I don't think we've had any problems with this being our slow path yet, but this may change in the future.

Yeah. IIRC the bo list create is typically < 10% of our kernel time for radv.

Thanks a lot for all your input.

Regards,
David Zhou

Quoted 1 line
Quoted 1 line

Dave.

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#51 lore raw
In reply to a message not in the archive

I agree with you about the spinlock. You seem to be good at this.

It's always good to do measurements to validate that a code change improves something, especially when the code size and code complexity has to be increased. A CPU profiler such as sysprof can show you improvements on the order of 1/10000th = 0.01% if you record enough samples. Sometimes you have to un-inline a function to make it visible there. If you see a function that takes 0.3% of CPU time and you optimize it down to 0.1% using the profiler as the measurement tool, you have evidence that the improvement is there and nobody can reject the idea anymore. It also proves that the code size increase is worth it. It's always "added code size and loss of simplicity" vs benefit. It's a transaction. You trade one for the other. You lose something to get something else. OK, we know the code complexity. Now, what's the benefit? Can you do some measurements? The accuracy of 1/10000th should be enough for anybody.

I know the feeling when you spend many days working on something, adding 100s or 1000s of lines of code, solving many problems to get there and increasing code complexity significantly, and then you do the measurement and it doesn't improve anything. I know the feeling very well. It sucks. The frustration comes from the investment of time and getting no return on the investment. Many frustrations in life are like that.

Marek

Quoted 31 lines · axie

Hi Marek,

So do you agree that spinlock disables CPU preemption, contrary to your original idea?

If you have new reason that this patch does not improve, please speak out.

Many patches in GPU driver aim at improving performance and power efficiency. Does most patches submitted in AMDGPU requires a benchmarking first?

If all developers are required to always answer your questions when code review, I am afraid that most open source community developers cannot meet that requirement and stop working on AMDGPU.

To improve performance, there are many bottlenecks to clear. When the last several bottlenecks are clear, the performance will show faster more significantly.

My pass profiling experience told me that clearing a lock can improve performance for some driver like 0.3% to much bigger percentage. It depends on many factors, even depends on the application itself.

This is not the first bottleneck fixed. This is surely not the last one.

Quoted 140 lines

Thanks,

Alex Bin

Quoted 132 lines · Marek Olšák

That's all nice, but does it improve performance? Have you been able to measure some performance difference with that code? Were you targeting a specific inefficiency you had seen e.g. with a CPU profiler?

Marek

Quoted 123 lines · axie

To clarify, local IRQ is disabled by calling raw_local_irq_save(flags);

Function __lock_acquire double checks that the local IRQ is really disabled.

Quoted 114 lines · axie

Hi Marek,

Spin lock and spin unlock is fast. But it is not so fast compared with atomic, which is a single CPU instruction in x86.

1. spinlock does NOT allow preemption at local CPU. Let us have a look at
how spin lock was implemented.

static inline void __raw_spin_lock(raw_spinlock_t *lock)

{
     preempt_disable(); <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<--This is
memory barrier operation too.
     spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
     LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
}

2. A function __lock_acquire called by spinlock. The function is so long
that I would not attach all of it here.

There is atomic operation inside and 12 meta data updates and 14 if statements and it calls quite some other functions.

Note that it disable IRQ...

static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
int trylock, int read, int check, int hardirqs_off, struct lockdep_map *nest_lock, unsigned long ip, int references, int pin_count)

{
     struct task_struct *curr = current;
     struct lock_class *class = NULL;
     struct held_lock *hlock;
     unsigned int depth;
     int chain_head = 0;
     int class_idx;
     u64 chain_key;

     if (unlikely(!debug_locks))
         return 0;

     /*
      * Lockdep should run with IRQs disabled, otherwise we could
      * get an interrupt which would want to take locks, which would
      * end up in lockdep and have you got a head-ache already?
      */
     if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) <<<<<<<<<<<<<<<Disable
IRQ
         return 0;

....

3. Another function called by spinlock in a higher level:

void lock_acquire(struct lockdep_map *lock, unsigned int subclass,

int trylock, int read, int check, struct lockdep_map *nest_lock, unsigned long ip)

{
     unsigned long flags;

     if (unlikely(current->lockdep_recursion))
         return;

     raw_local_irq_save(flags);
     check_flags(flags);

     current->lockdep_recursion = 1;
     trace_lock_acquire(lock, subclass, trylock, read, check, nest_lock,
ip);
     __lock_acquire(lock, subclass, trylock, read, check,
                irqs_disabled_flags(flags), nest_lock, ip, 0, 0);
     current->lockdep_recursion = 0;
     raw_local_irq_restore(flags);
}

Thanks,

Alex Bin

Quoted 25 lines · Marek Olšák
Quoted 10 lines · AlexBin

Hi Christian,

In fact, the change from spinlock to atomic is quite painful. When I started, I thought it was easy but later I found there might be race condition here and there. Now I think the change looks more robust. In kernel source, there are several other drivers used the same trick.

Quoted 4 lines

On the other hand, I think the logic itself might be optimized considering
the locking. But I had spent quite some effort to maintain original logic.

It seems quite complicated and I don't know if there is any performance benefit. Spinlocks are nice because they allow preemption.

It would be more interesting to merge the CS and BO_LIST ioctls into one.

Marek

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): CAAxE2A6Tt1JzLCDr4vM1iKASpWGFncqrXkL75==1Zd5wJq8xTA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#53 lore raw

The key point here is while optimizing this is nice the much bigger pile is the locking done for each BO.

In other words even when we optimize all the other locks involved into atomics or RCU, the BO reservation lock will still dominate everything.

One possible solution to this would be per process resources like I suggested multiple times now.

Christian.

Am 23.06.2017 um 13:37 schrieb Marek Olšák:

Quoted 25 lines

I agree with you about the spinlock. You seem to be good at this.

It's always good to do measurements to validate that a code change improves something, especially when the code size and code complexity has to be increased. A CPU profiler such as sysprof can show you improvements on the order of 1/10000th = 0.01% if you record enough samples. Sometimes you have to un-inline a function to make it visible there. If you see a function that takes 0.3% of CPU time and you optimize it down to 0.1% using the profiler as the measurement tool, you have evidence that the improvement is there and nobody can reject the idea anymore. It also proves that the code size increase is worth it. It's always "added code size and loss of simplicity" vs benefit. It's a transaction. You trade one for the other. You lose something to get something else. OK, we know the code complexity. Now, what's the benefit? Can you do some measurements? The accuracy of 1/10000th should be enough for anybody.

I know the feeling when you spend many days working on something, adding 100s or 1000s of lines of code, solving many problems to get there and increasing code complexity significantly, and then you do the measurement and it doesn't improve anything. I know the feeling very well. It sucks. The frustration comes from the investment of time and getting no return on the investment. Many frustrations in life are like that.

Quoted 167 lines

Marek

Quoted 163 lines · axie

Hi Marek,

So do you agree that spinlock disables CPU preemption, contrary to your original idea?

If you have new reason that this patch does not improve, please speak out.

Many patches in GPU driver aim at improving performance and power efficiency. Does most patches submitted in AMDGPU requires a benchmarking first?

If all developers are required to always answer your questions when code review, I am afraid that most open source community developers cannot meet that requirement and stop working on AMDGPU.

To improve performance, there are many bottlenecks to clear. When the last several bottlenecks are clear, the performance will show faster more significantly.

My pass profiling experience told me that clearing a lock can improve performance for some driver like 0.3% to much bigger percentage. It depends on many factors, even depends on the application itself.

This is not the first bottleneck fixed. This is surely not the last one.

Thanks,

Alex Bin

Quoted 125 lines · Marek Olšák

That's all nice, but does it improve performance? Have you been able to measure some performance difference with that code? Were you targeting a specific inefficiency you had seen e.g. with a CPU profiler?

Marek

Quoted 117 lines · axie

To clarify, local IRQ is disabled by calling raw_local_irq_save(flags);

Function __lock_acquire double checks that the local IRQ is really disabled.

Quoted 109 lines · axie

Hi Marek,

Spin lock and spin unlock is fast. But it is not so fast compared with atomic, which is a single CPU instruction in x86.

1. spinlock does NOT allow preemption at local CPU. Let us have a look at
how spin lock was implemented.

static inline void __raw_spin_lock(raw_spinlock_t *lock)

{
      preempt_disable(); <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<--This is
memory barrier operation too.
      spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
      LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
}

2. A function __lock_acquire called by spinlock. The function is so long
that I would not attach all of it here.

There is atomic operation inside and 12 meta data updates and 14 if statements and it calls quite some other functions.

Note that it disable IRQ...

static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
int trylock, int read, int check, int hardirqs_off, struct lockdep_map *nest_lock, unsigned long ip, int references, int pin_count)

{
      struct task_struct *curr = current;
      struct lock_class *class = NULL;
      struct held_lock *hlock;
      unsigned int depth;
      int chain_head = 0;
      int class_idx;
      u64 chain_key;

      if (unlikely(!debug_locks))
          return 0;

      /*
       * Lockdep should run with IRQs disabled, otherwise we could
       * get an interrupt which would want to take locks, which would
       * end up in lockdep and have you got a head-ache already?
       */
      if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) <<<<<<<<<<<<<<<Disable
IRQ
          return 0;

....

3. Another function called by spinlock in a higher level:

void lock_acquire(struct lockdep_map *lock, unsigned int subclass,

int trylock, int read, int check, struct lockdep_map *nest_lock, unsigned long ip)

{
      unsigned long flags;

      if (unlikely(current->lockdep_recursion))
          return;

      raw_local_irq_save(flags);
      check_flags(flags);

      current->lockdep_recursion = 1;
      trace_lock_acquire(lock, subclass, trylock, read, check, nest_lock,
ip);
      __lock_acquire(lock, subclass, trylock, read, check,
                 irqs_disabled_flags(flags), nest_lock, ip, 0, 0);
      current->lockdep_recursion = 0;
      raw_local_irq_restore(flags);
}

Thanks,

Alex Bin

Quoted 22 lines · Marek Olšák
Quoted 9 lines · AlexBin

Hi Christian,

In fact, the change from spinlock to atomic is quite painful. When I started, I thought it was easy but later I found there might be race condition here and there. Now I think the change looks more robust. In kernel source, there are several other drivers used the same trick.

Quoted 4 lines

On the other hand, I think the logic itself might be optimized considering
the locking. But I had spent quite some effort to maintain original logic.

It seems quite complicated and I don't know if there is any performance benefit. Spinlocks are nice because they allow preemption.

It would be more interesting to merge the CS and BO_LIST ioctls into one.

Marek

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 963f4d3d-dc46-2279-509f-b475c5ec94ee-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

#55 lore raw

Hi Christian,

I agree with you. On the other hand, after you optimize the BO reservation lock, other locks still need optimization, right?

1. Locking itself is not cheap.

2. Waiting in lock is even more expensive.

Thanks,

Alex Bin Xie

Quoted 206 lines · Christian König

The key point here is while optimizing this is nice the much bigger pile is the locking done for each BO.

In other words even when we optimize all the other locks involved into atomics or RCU, the BO reservation lock will still dominate everything.

One possible solution to this would be per process resources like I suggested multiple times now.

Christian.

Am 23.06.2017 um 13:37 schrieb Marek Olšák:

Quoted 196 lines

I agree with you about the spinlock. You seem to be good at this.

It's always good to do measurements to validate that a code change improves something, especially when the code size and code complexity has to be increased. A CPU profiler such as sysprof can show you improvements on the order of 1/10000th = 0.01% if you record enough samples. Sometimes you have to un-inline a function to make it visible there. If you see a function that takes 0.3% of CPU time and you optimize it down to 0.1% using the profiler as the measurement tool, you have evidence that the improvement is there and nobody can reject the idea anymore. It also proves that the code size increase is worth it. It's always "added code size and loss of simplicity" vs benefit. It's a transaction. You trade one for the other. You lose something to get something else. OK, we know the code complexity. Now, what's the benefit? Can you do some measurements? The accuracy of 1/10000th should be enough for anybody.

I know the feeling when you spend many days working on something, adding 100s or 1000s of lines of code, solving many problems to get there and increasing code complexity significantly, and then you do the measurement and it doesn't improve anything. I know the feeling very well. It sucks. The frustration comes from the investment of time and getting no return on the investment. Many frustrations in life are like that.

Marek

Quoted 167 lines · axie

Hi Marek,

So do you agree that spinlock disables CPU preemption, contrary to your original idea?

If you have new reason that this patch does not improve, please speak out.

Many patches in GPU driver aim at improving performance and power efficiency. Does most patches submitted in AMDGPU requires a benchmarking first?

If all developers are required to always answer your questions when code review, I am afraid that most open source community developers cannot meet that requirement and stop working on AMDGPU.

To improve performance, there are many bottlenecks to clear. When the last several bottlenecks are clear, the performance will show faster more significantly.

My pass profiling experience told me that clearing a lock can improve performance for some driver like 0.3% to much bigger percentage. It depends on many factors, even depends on the application itself.

This is not the first bottleneck fixed. This is surely not the last one.

Thanks,

Alex Bin

Quoted 129 lines · Marek Olšák

That's all nice, but does it improve performance? Have you been able to measure some performance difference with that code? Were you targeting a specific inefficiency you had seen e.g. with a CPU profiler?

Marek

Quoted 121 lines · axie

To clarify, local IRQ is disabled by calling raw_local_irq_save(flags);

Function __lock_acquire double checks that the local IRQ is really disabled.

Quoted 113 lines · axie

Hi Marek,

Spin lock and spin unlock is fast. But it is not so fast compared with atomic, which is a single CPU instruction in x86.

1. spinlock does NOT allow preemption at local CPU. Let us have a look at
how spin lock was implemented.

static inline void __raw_spin_lock(raw_spinlock_t *lock)

{
      preempt_disable(); 
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<--This is
memory barrier operation too.
      spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
      LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
}

2. A function __lock_acquire called by spinlock. The function is so long
that I would not attach all of it here.

There is atomic operation inside and 12 meta data updates and 14 if statements and it calls quite some other functions.

Note that it disable IRQ...

static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
int trylock, int read, int check, int hardirqs_off, struct lockdep_map *nest_lock, unsigned long ip, int references, int pin_count)

{
      struct task_struct *curr = current;
      struct lock_class *class = NULL;
      struct held_lock *hlock;
      unsigned int depth;
      int chain_head = 0;
      int class_idx;
      u64 chain_key;

      if (unlikely(!debug_locks))
          return 0;

      /*
       * Lockdep should run with IRQs disabled, otherwise we could
       * get an interrupt which would want to take locks, which 
would
       * end up in lockdep and have you got a head-ache already?
       */
      if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) 
<<<<<<<<<<<<<<<Disable
IRQ
          return 0;

....

3. Another function called by spinlock in a higher level:

void lock_acquire(struct lockdep_map *lock, unsigned int subclass,

int trylock, int read, int check, struct lockdep_map *nest_lock, unsigned long ip)

{
      unsigned long flags;

      if (unlikely(current->lockdep_recursion))
          return;

      raw_local_irq_save(flags);
      check_flags(flags);

      current->lockdep_recursion = 1;
      trace_lock_acquire(lock, subclass, trylock, read, check, 
nest_lock,
ip);
      __lock_acquire(lock, subclass, trylock, read, check,
                 irqs_disabled_flags(flags), nest_lock, ip, 0, 0);
      current->lockdep_recursion = 0;
      raw_local_irq_restore(flags);
}

Thanks,

Alex Bin

Quoted 22 lines · Marek Olšák
Quoted 9 lines · AlexBin

Hi Christian,

In fact, the change from spinlock to atomic is quite painful. When I started, I thought it was easy but later I found there might be race condition here and there. Now I think the change looks more robust. In kernel source, there are several other drivers used the same trick.

Quoted 4 lines

On the other hand, I think the logic itself might be optimized considering
the locking. But I had spent quite some effort to maintain original logic.

It seems quite complicated and I don't know if there is any performance benefit. Spinlocks are nice because they allow preemption.

It would be more interesting to merge the CS and BO_LIST ioctls into one.

Marek

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 3d4f3204-6893-3d68-23ea-b309ace33740-5C7GfCeVMHo@public.gmane.org

#57 lore raw
Quoted 1 line

On the other hand, after you optimize the BO reservation lock, other locks still need optimization, right?

In theory yes, in practice no.

There are just way other things we should tackle before taking care of removing any locks that we probably never get to that point even with more manpower.

Christian.

Am 23.06.2017 um 15:55 schrieb axie:

Quoted 220 lines

Hi Christian,

I agree with you. On the other hand, after you optimize the BO reservation lock, other locks still need optimization, right?

1. Locking itself is not cheap.

2. Waiting in lock is even more expensive.

Thanks,

Alex Bin Xie

Quoted 203 lines · Christian König

The key point here is while optimizing this is nice the much bigger pile is the locking done for each BO.

In other words even when we optimize all the other locks involved into atomics or RCU, the BO reservation lock will still dominate everything.

One possible solution to this would be per process resources like I suggested multiple times now.

Christian.

Am 23.06.2017 um 13:37 schrieb Marek Olšák:

Quoted 194 lines

I agree with you about the spinlock. You seem to be good at this.

It's always good to do measurements to validate that a code change improves something, especially when the code size and code complexity has to be increased. A CPU profiler such as sysprof can show you improvements on the order of 1/10000th = 0.01% if you record enough samples. Sometimes you have to un-inline a function to make it visible there. If you see a function that takes 0.3% of CPU time and you optimize it down to 0.1% using the profiler as the measurement tool, you have evidence that the improvement is there and nobody can reject the idea anymore. It also proves that the code size increase is worth it. It's always "added code size and loss of simplicity" vs benefit. It's a transaction. You trade one for the other. You lose something to get something else. OK, we know the code complexity. Now, what's the benefit? Can you do some measurements? The accuracy of 1/10000th should be enough for anybody.

I know the feeling when you spend many days working on something, adding 100s or 1000s of lines of code, solving many problems to get there and increasing code complexity significantly, and then you do the measurement and it doesn't improve anything. I know the feeling very well. It sucks. The frustration comes from the investment of time and getting no return on the investment. Many frustrations in life are like that.

Marek

Quoted 165 lines · axie

Hi Marek,

So do you agree that spinlock disables CPU preemption, contrary to your original idea?

If you have new reason that this patch does not improve, please speak out.

Many patches in GPU driver aim at improving performance and power efficiency. Does most patches submitted in AMDGPU requires a benchmarking first?

If all developers are required to always answer your questions when code review, I am afraid that most open source community developers cannot meet that requirement and stop working on AMDGPU.

To improve performance, there are many bottlenecks to clear. When the last several bottlenecks are clear, the performance will show faster more significantly.

My pass profiling experience told me that clearing a lock can improve performance for some driver like 0.3% to much bigger percentage. It depends on many factors, even depends on the application itself.

This is not the first bottleneck fixed. This is surely not the last one.

Thanks,

Alex Bin

Quoted 127 lines · Marek Olšák

That's all nice, but does it improve performance? Have you been able to measure some performance difference with that code? Were you targeting a specific inefficiency you had seen e.g. with a CPU profiler?

Marek

Quoted 119 lines · axie

To clarify, local IRQ is disabled by calling raw_local_irq_save(flags);

Function __lock_acquire double checks that the local IRQ is really disabled.

Quoted 111 lines · axie

Hi Marek,

Spin lock and spin unlock is fast. But it is not so fast compared with atomic, which is a single CPU instruction in x86.

1. spinlock does NOT allow preemption at local CPU. Let us have a look at
how spin lock was implemented.

static inline void __raw_spin_lock(raw_spinlock_t *lock)

{
      preempt_disable(); 
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<--This is
memory barrier operation too.
      spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
      LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
}

2. A function __lock_acquire called by spinlock. The function is so long
that I would not attach all of it here.

There is atomic operation inside and 12 meta data updates and 14 if statements and it calls quite some other functions.

Note that it disable IRQ...

static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
int trylock, int read, int check, int hardirqs_off, struct lockdep_map *nest_lock, unsigned long ip, int references, int pin_count)

{
      struct task_struct *curr = current;
      struct lock_class *class = NULL;
      struct held_lock *hlock;
      unsigned int depth;
      int chain_head = 0;
      int class_idx;
      u64 chain_key;

      if (unlikely(!debug_locks))
          return 0;

      /*
       * Lockdep should run with IRQs disabled, otherwise we could
       * get an interrupt which would want to take locks, which 
would
       * end up in lockdep and have you got a head-ache already?
       */
      if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) 
<<<<<<<<<<<<<<<Disable
IRQ
          return 0;

....

3. Another function called by spinlock in a higher level:

void lock_acquire(struct lockdep_map *lock, unsigned int subclass,

int trylock, int read, int check, struct lockdep_map *nest_lock, unsigned long ip)

{
      unsigned long flags;

      if (unlikely(current->lockdep_recursion))
          return;

      raw_local_irq_save(flags);
      check_flags(flags);

      current->lockdep_recursion = 1;
      trace_lock_acquire(lock, subclass, trylock, read, check, 
nest_lock,
ip);
      __lock_acquire(lock, subclass, trylock, read, check,
                 irqs_disabled_flags(flags), nest_lock, ip, 0, 0);
      current->lockdep_recursion = 0;
      raw_local_irq_restore(flags);
}

Thanks,

Alex Bin

Quoted 21 lines · Marek Olšák
Quoted 9 lines · AlexBin

Hi Christian,

In fact, the change from spinlock to atomic is quite painful. When I started, I thought it was easy but later I found there might be race condition here and there. Now I think the change looks more robust. In kernel source, there are several other drivers used the same trick.

Quoted 4 lines

On the other hand, I think the logic itself might be optimized considering
the locking. But I had spent quite some effort to maintain original logic.

It seems quite complicated and I don't know if there is any performance benefit. Spinlocks are nice because they allow preemption.

It would be more interesting to merge the CS and BO_LIST ioctls into one.

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

Marek

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#58 lore raw
In reply to a message not in the archive
Quoted 8 lines · Christian König

The key point here is while optimizing this is nice the much bigger pile is the locking done for each BO.

In other words even when we optimize all the other locks involved into atomics or RCU, the BO reservation lock will still dominate everything.

One possible solution to this would be per process resources like I suggested multiple times now.

Mesa can set a per-process resource flag on all resources except displayable ones. The question is, would it help if an IB contained 1000 per-process resources and 1-2 inter-process sharable?

Marek

Quoted 215 lines

Christian.

Am 23.06.2017 um 13:37 schrieb Marek Olšák:

Quoted 209 lines

I agree with you about the spinlock. You seem to be good at this.

It's always good to do measurements to validate that a code change improves something, especially when the code size and code complexity has to be increased. A CPU profiler such as sysprof can show you improvements on the order of 1/10000th = 0.01% if you record enough samples. Sometimes you have to un-inline a function to make it visible there. If you see a function that takes 0.3% of CPU time and you optimize it down to 0.1% using the profiler as the measurement tool, you have evidence that the improvement is there and nobody can reject the idea anymore. It also proves that the code size increase is worth it. It's always "added code size and loss of simplicity" vs benefit. It's a transaction. You trade one for the other. You lose something to get something else. OK, we know the code complexity. Now, what's the benefit? Can you do some measurements? The accuracy of 1/10000th should be enough for anybody.

I know the feeling when you spend many days working on something, adding 100s or 1000s of lines of code, solving many problems to get there and increasing code complexity significantly, and then you do the measurement and it doesn't improve anything. I know the feeling very well. It sucks. The frustration comes from the investment of time and getting no return on the investment. Many frustrations in life are like that.

Marek

Quoted 179 lines · axie

Hi Marek,

So do you agree that spinlock disables CPU preemption, contrary to your original idea?

If you have new reason that this patch does not improve, please speak out.

Many patches in GPU driver aim at improving performance and power efficiency. Does most patches submitted in AMDGPU requires a benchmarking first?

If all developers are required to always answer your questions when code review, I am afraid that most open source community developers cannot meet
that requirement and stop working on AMDGPU.

To improve performance, there are many bottlenecks to clear. When the last
several bottlenecks are clear, the performance will show faster more significantly.

My pass profiling experience told me that clearing a lock can improve performance for some driver like 0.3% to much bigger percentage. It depends
on many factors, even depends on the application itself.

This is not the first bottleneck fixed. This is surely not the last one.

Thanks,

Alex Bin

Quoted 136 lines · Marek Olšák

That's all nice, but does it improve performance? Have you been able to measure some performance difference with that code? Were you targeting a specific inefficiency you had seen e.g. with a CPU profiler?

Marek

Quoted 127 lines · axie

To clarify, local IRQ is disabled by calling raw_local_irq_save(flags);

Function __lock_acquire double checks that the local IRQ is really disabled.

Quoted 118 lines · axie

Hi Marek,

Spin lock and spin unlock is fast. But it is not so fast compared with atomic, which is a single CPU instruction in x86.

1. spinlock does NOT allow preemption at local CPU. Let us have a look at
how spin lock was implemented.

static inline void __raw_spin_lock(raw_spinlock_t *lock)

{
      preempt_disable(); <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<--This is
memory barrier operation too.
      spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
      LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
}

2. A function __lock_acquire called by spinlock. The function is so long
that I would not attach all of it here.

There is atomic operation inside and 12 meta data updates and 14 if statements and it calls quite some other functions.

Note that it disable IRQ...

static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
int trylock, int read, int check, int hardirqs_off, struct lockdep_map *nest_lock, unsigned long ip, int references, int pin_count)

{
      struct task_struct *curr = current;
      struct lock_class *class = NULL;
      struct held_lock *hlock;
      unsigned int depth;
      int chain_head = 0;
      int class_idx;
      u64 chain_key;

      if (unlikely(!debug_locks))
          return 0;

      /*
       * Lockdep should run with IRQs disabled, otherwise we could
       * get an interrupt which would want to take locks, which would
       * end up in lockdep and have you got a head-ache already?
       */
      if (DEBUG_LOCKS_WARN_ON(!irqs_disabled()))
<<<<<<<<<<<<<<<Disable
IRQ
          return 0;

....

3. Another function called by spinlock in a higher level:

void lock_acquire(struct lockdep_map *lock, unsigned int subclass,

int trylock, int read, int check, struct lockdep_map *nest_lock, unsigned long ip)

{
      unsigned long flags;

      if (unlikely(current->lockdep_recursion))
          return;

      raw_local_irq_save(flags);
      check_flags(flags);

      current->lockdep_recursion = 1;
      trace_lock_acquire(lock, subclass, trylock, read, check,
nest_lock,
ip);
      __lock_acquire(lock, subclass, trylock, read, check,
                 irqs_disabled_flags(flags), nest_lock, ip, 0, 0);
      current->lockdep_recursion = 0;
      raw_local_irq_restore(flags);
}

Thanks,

Alex Bin

Quoted 27 lines · Marek Olšák
Quoted 11 lines · AlexBin

Hi Christian,

In fact, the change from spinlock to atomic is quite painful. When I started, I thought it was easy but later I found there might be race condition here and there. Now I think the change looks more robust. In
kernel source, there are several other drivers used the same trick.

Quoted 4 lines

On the other hand, I think the logic itself might be optimized considering
the locking. But I had spent quite some effort to maintain original logic.

It seems quite complicated and I don't know if there is any performance benefit. Spinlocks are nice because they allow preemption.

It would be more interesting to merge the CS and BO_LIST ioctls into one.

Marek

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): CAAxE2A7kFusP3=-FcVWdSnGRvaFwL8kZ17BH4uKJ3z044S9XFQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#60 lore raw

Am 24.06.2017 um 02:27 schrieb Marek Olšák:

Quoted 5 lines
Quoted 3 lines · Christian König

The key point here is while optimizing this is nice the much bigger pile is the locking done for each BO.

Quoted 8 lines
Quoted 5 lines

In other words even when we optimize all the other locks involved into atomics or RCU, the BO reservation lock will still dominate everything.

One possible solution to this would be per process resources like I suggested multiple times now.

Mesa can set a per-process resource flag on all resources except displayable ones. The question is, would it help if an IB contained 1000 per-process resources and 1-2 inter-process sharable?

Yeah, absolutely.

See the 1000 per-process shareable resources only need to be revalidated when one of them was evicted. This would easily reduce the per BO CS overhead to 1/1000 of what it is now.

The crux is that you can't share those resources with other process and since you don't specify them on a per CS basis swapping things in and out on per CS basis like we do for all the Unigine benchmarks won't work any more.

Everything would be statically allocated, that might not be ideal either.

Regards,
Christian.

Quoted 207 lines

Marek

Quoted 204 lines

Christian.

Am 23.06.2017 um 13:37 schrieb Marek Olšák:

Quoted 200 lines

I agree with you about the spinlock. You seem to be good at this.

It's always good to do measurements to validate that a code change improves something, especially when the code size and code complexity has to be increased. A CPU profiler such as sysprof can show you improvements on the order of 1/10000th = 0.01% if you record enough samples. Sometimes you have to un-inline a function to make it visible there. If you see a function that takes 0.3% of CPU time and you optimize it down to 0.1% using the profiler as the measurement tool, you have evidence that the improvement is there and nobody can reject the idea anymore. It also proves that the code size increase is worth it. It's always "added code size and loss of simplicity" vs benefit. It's a transaction. You trade one for the other. You lose something to get something else. OK, we know the code complexity. Now, what's the benefit? Can you do some measurements? The accuracy of 1/10000th should be enough for anybody.

I know the feeling when you spend many days working on something, adding 100s or 1000s of lines of code, solving many problems to get there and increasing code complexity significantly, and then you do the measurement and it doesn't improve anything. I know the feeling very well. It sucks. The frustration comes from the investment of time and getting no return on the investment. Many frustrations in life are like that.

Marek

Quoted 171 lines · axie

Hi Marek,

So do you agree that spinlock disables CPU preemption, contrary to your original idea?

If you have new reason that this patch does not improve, please speak out.

Many patches in GPU driver aim at improving performance and power efficiency. Does most patches submitted in AMDGPU requires a benchmarking first?

If all developers are required to always answer your questions when code review, I am afraid that most open source community developers cannot meet
that requirement and stop working on AMDGPU.

To improve performance, there are many bottlenecks to clear. When the last
several bottlenecks are clear, the performance will show faster more significantly.

My pass profiling experience told me that clearing a lock can improve performance for some driver like 0.3% to much bigger percentage. It depends
on many factors, even depends on the application itself.

This is not the first bottleneck fixed. This is surely not the last one.

Thanks,

Alex Bin

Quoted 129 lines · Marek Olšák

That's all nice, but does it improve performance? Have you been able to measure some performance difference with that code? Were you targeting a specific inefficiency you had seen e.g. with a CPU profiler?

Marek

Quoted 121 lines · axie

To clarify, local IRQ is disabled by calling raw_local_irq_save(flags);

Function __lock_acquire double checks that the local IRQ is really disabled.

Quoted 113 lines · axie

Hi Marek,

Spin lock and spin unlock is fast. But it is not so fast compared with atomic, which is a single CPU instruction in x86.

1. spinlock does NOT allow preemption at local CPU. Let us have a look at
how spin lock was implemented.

static inline void __raw_spin_lock(raw_spinlock_t *lock)

{
       preempt_disable(); <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<--This is
memory barrier operation too.
       spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
       LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
}

2. A function __lock_acquire called by spinlock. The function is so long
that I would not attach all of it here.

There is atomic operation inside and 12 meta data updates and 14 if statements and it calls quite some other functions.

Note that it disable IRQ...

static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
int trylock, int read, int check, int hardirqs_off, struct lockdep_map *nest_lock, unsigned long ip, int references, int pin_count)

{
       struct task_struct *curr = current;
       struct lock_class *class = NULL;
       struct held_lock *hlock;
       unsigned int depth;
       int chain_head = 0;
       int class_idx;
       u64 chain_key;

       if (unlikely(!debug_locks))
           return 0;

       /*
        * Lockdep should run with IRQs disabled, otherwise we could
        * get an interrupt which would want to take locks, which would
        * end up in lockdep and have you got a head-ache already?
        */
       if (DEBUG_LOCKS_WARN_ON(!irqs_disabled()))
<<<<<<<<<<<<<<<Disable
IRQ
           return 0;

....

3. Another function called by spinlock in a higher level:

void lock_acquire(struct lockdep_map *lock, unsigned int subclass,

int trylock, int read, int check, struct lockdep_map *nest_lock, unsigned long ip)

{
       unsigned long flags;

       if (unlikely(current->lockdep_recursion))
           return;

       raw_local_irq_save(flags);
       check_flags(flags);

       current->lockdep_recursion = 1;
       trace_lock_acquire(lock, subclass, trylock, read, check,
nest_lock,
ip);
       __lock_acquire(lock, subclass, trylock, read, check,
                  irqs_disabled_flags(flags), nest_lock, ip, 0, 0);
       current->lockdep_recursion = 0;
       raw_local_irq_restore(flags);
}

Thanks,

Alex Bin

Quoted 24 lines · Marek Olšák
Quoted 10 lines · AlexBin

Hi Christian,

In fact, the change from spinlock to atomic is quite painful. When I started, I thought it was easy but later I found there might be race condition here and there. Now I think the change looks more robust. In
kernel source, there are several other drivers used the same trick.

Quoted 4 lines

On the other hand, I think the logic itself might be optimized considering
the locking. But I had spent quite some effort to maintain original logic.

It seems quite complicated and I don't know if there is any performance benefit. Spinlocks are nice because they allow preemption.

It would be more interesting to merge the CS and BO_LIST ioctls into one.

Marek

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#61 lore raw
In reply to a message not in the archive

Hi Marek,

I understand you spent time on your original logic too. I really don't understand why you talked about pain if somebody can improve it.

To reduce the pain, now I am seriously considering dropping this patch. But please read on before you conclude. Let us treat open source software development a fun.

Same trick like this patch could be found in open source Intel GPU driver and xfs.

Talking about code size. You will be surprised if you really calculate it.

For function amdgpu_cs_get_threshold_for_moves:

New code:  It is 2 more loops and 2 more ifs.
Old code:  2 spinlock inline function. spin_lock can expand to 5 function calls and one if. One function call contains 7 parameters.

spin_unlock can expand to 4 function calls.

By the way, you can config Linux kernel to disable some spinlock macro expansion. But I don't think people really do that.

In function amdgpu_cs_report_moved_bytes

New code:  zero
Old code:  2 spinlock inline function.

In Total:

New code:  It is 2 more loops and 2 more ifs. Maybe there are one or two other tiny things.
Old code:  4 spinlock inline function. They are expanded to 18 function calls. Among them, two function calls each contain 7 parameters.

Please think about it. Are you still sure that the new code make code size bigger? Now what is the next problem of the new change?

I agree that it is more difficult to understand the new code. But if you get used to it. It is not so difficult in deed. Just one loop to retry. Human are much smarter than this logic. Compared with how hardware engineers went to extreme to optimize logics and design with optimization from day 1, my tiny new logic is really nothing.

I said that removing a lock can improve 0.3% or even bigger for some driver. I did not say it was AMDGPU. My tiny improvement may not be so obvious in this big driver for the time being.

I will give you the privilege to make a final decision, for example, you can even delay it for future if you don't want to make a decision now. Please be happy.

Thanks,
Alex Bin Xie

Quoted 25 lines · Marek Olšák

I agree with you about the spinlock. You seem to be good at this.

It's always good to do measurements to validate that a code change improves something, especially when the code size and code complexity has to be increased. A CPU profiler such as sysprof can show you improvements on the order of 1/10000th = 0.01% if you record enough samples. Sometimes you have to un-inline a function to make it visible there. If you see a function that takes 0.3% of CPU time and you optimize it down to 0.1% using the profiler as the measurement tool, you have evidence that the improvement is there and nobody can reject the idea anymore. It also proves that the code size increase is worth it. It's always "added code size and loss of simplicity" vs benefit. It's a transaction. You trade one for the other. You lose something to get something else. OK, we know the code complexity. Now, what's the benefit? Can you do some measurements? The accuracy of 1/10000th should be enough for anybody.

I know the feeling when you spend many days working on something, adding 100s or 1000s of lines of code, solving many problems to get there and increasing code complexity significantly, and then you do the measurement and it doesn't improve anything. I know the feeling very well. It sucks. The frustration comes from the investment of time and getting no return on the investment. Many frustrations in life are like that.

Quoted 167 lines

Marek

Quoted 163 lines · axie

Hi Marek,

So do you agree that spinlock disables CPU preemption, contrary to your original idea?

If you have new reason that this patch does not improve, please speak out.

Many patches in GPU driver aim at improving performance and power efficiency. Does most patches submitted in AMDGPU requires a benchmarking first?

If all developers are required to always answer your questions when code review, I am afraid that most open source community developers cannot meet that requirement and stop working on AMDGPU.

To improve performance, there are many bottlenecks to clear. When the last several bottlenecks are clear, the performance will show faster more significantly.

My pass profiling experience told me that clearing a lock can improve performance for some driver like 0.3% to much bigger percentage. It depends on many factors, even depends on the application itself.

This is not the first bottleneck fixed. This is surely not the last one.

Thanks,

Alex Bin

Quoted 125 lines · Marek Olšák

That's all nice, but does it improve performance? Have you been able to measure some performance difference with that code? Were you targeting a specific inefficiency you had seen e.g. with a CPU profiler?

Marek

Quoted 117 lines · axie

To clarify, local IRQ is disabled by calling raw_local_irq_save(flags);

Function __lock_acquire double checks that the local IRQ is really disabled.

Quoted 109 lines · axie

Hi Marek,

Spin lock and spin unlock is fast. But it is not so fast compared with atomic, which is a single CPU instruction in x86.

1. spinlock does NOT allow preemption at local CPU. Let us have a look at
how spin lock was implemented.

static inline void __raw_spin_lock(raw_spinlock_t *lock)

{
      preempt_disable(); <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<--This is
memory barrier operation too.
      spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
      LOCK_CONTENDED(lock, do_raw_spin_trylock, do_raw_spin_lock);
}

2. A function __lock_acquire called by spinlock. The function is so long
that I would not attach all of it here.

There is atomic operation inside and 12 meta data updates and 14 if statements and it calls quite some other functions.

Note that it disable IRQ...

static int __lock_acquire(struct lockdep_map *lock, unsigned int subclass,
int trylock, int read, int check, int hardirqs_off, struct lockdep_map *nest_lock, unsigned long ip, int references, int pin_count)

{
      struct task_struct *curr = current;
      struct lock_class *class = NULL;
      struct held_lock *hlock;
      unsigned int depth;
      int chain_head = 0;
      int class_idx;
      u64 chain_key;

      if (unlikely(!debug_locks))
          return 0;

      /*
       * Lockdep should run with IRQs disabled, otherwise we could
       * get an interrupt which would want to take locks, which would
       * end up in lockdep and have you got a head-ache already?
       */
      if (DEBUG_LOCKS_WARN_ON(!irqs_disabled())) <<<<<<<<<<<<<<<Disable
IRQ
          return 0;

....

3. Another function called by spinlock in a higher level:

void lock_acquire(struct lockdep_map *lock, unsigned int subclass,

int trylock, int read, int check, struct lockdep_map *nest_lock, unsigned long ip)

{
      unsigned long flags;

      if (unlikely(current->lockdep_recursion))
          return;

      raw_local_irq_save(flags);
      check_flags(flags);

      current->lockdep_recursion = 1;
      trace_lock_acquire(lock, subclass, trylock, read, check, nest_lock,
ip);
      __lock_acquire(lock, subclass, trylock, read, check,
                 irqs_disabled_flags(flags), nest_lock, ip, 0, 0);
      current->lockdep_recursion = 0;
      raw_local_irq_restore(flags);
}

Thanks,

Alex Bin

Quoted 22 lines · Marek Olšák
Quoted 9 lines · AlexBin

Hi Christian,

In fact, the change from spinlock to atomic is quite painful. When I started, I thought it was easy but later I found there might be race condition here and there. Now I think the change looks more robust. In kernel source, there are several other drivers used the same trick.

Quoted 4 lines

On the other hand, I think the logic itself might be optimized considering
the locking. But I had spent quite some effort to maintain original logic.

It seems quite complicated and I don't know if there is any performance benefit. Spinlocks are nice because they allow preemption.

It would be more interesting to merge the CS and BO_LIST ioctls into one.

Marek

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 94b20fb6-3f81-da3c-eb44-7d2e49ff5c8b-5C7GfCeVMHo@public.gmane.org

#63 lore raw
Quoted 29 lines · axie

Hi Marek,

I understand you spent time on your original logic too. I really don't understand why you talked about pain if somebody can improve it.

To reduce the pain, now I am seriously considering dropping this patch. But please read on before you conclude. Let us treat open source software development a fun.

Same trick like this patch could be found in open source Intel GPU driver and xfs.

Talking about code size. You will be surprised if you really calculate it.

For function amdgpu_cs_get_threshold_for_moves:

New code:  It is 2 more loops and 2 more ifs.
Old code:  2 spinlock inline function. spin_lock can expand to 5 function

calls and one if. One function call contains 7 parameters. spin_unlock can expand to 4 function calls.

By the way, you can config Linux kernel to disable some spinlock macro expansion. But I don't think people really do that.

In function amdgpu_cs_report_moved_bytes

New code:  zero
Old code:  2 spinlock inline function.

In Total:

Quoted 7 lines

New code: It is 2 more loops and 2 more ifs. Maybe there are one or two other tiny things.
Old code: 4 spinlock inline function. They are expanded to 18 function calls. Among them, two function calls each contain 7 parameters.

Please think about it. Are you still sure that the new code make code size bigger? Now what is the next problem of the new change?

The code size means the size of source code, not binary. The fewer lines of amdgpu code that we need to get the job done, the better.

The next issue is the risk of breaking this already hard-to-test code.

Your logic would be OK if there was a measurable benefit even with the silliest microbenchmark you can find (and even if it were as low as 0.2% improvement). Without that, I can't accept it. Sorry.

Marek
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx