Skip to contentSkip to conversation

Performance regression in ast drm driver

DISCUSSION Jean Delvare 5 participants 12 messages Nov 1, 2018 – Nov 13, 2018

lore
Share & tools
lore
#0 lore raw

Hi David,

The following commit:

commit 7cf321d118a825c1541b43ca45294126fd474efa

Author: Dave Airlie
Date:   Mon Oct 24 15:37:48 2016 +1000

drm/drivers: add support for using the arch wc mapping API.

is causing a huge performance regression for the ast drm driver. In a text console, if I call "cat" on a large text file, it takes almost twice as much time to be displayed and scrolled completely.

Can you please check that the ast driver portion of that commit is both correct and complete?

Thanks,

Signature · 2 lines
Jean Delvare
SUSE L3 Support
#1 lore raw
Quoted 8 lines · Jean Delvare

Hi David,

The following commit:

commit 7cf321d118a825c1541b43ca45294126fd474efa

Author: Dave Airlie
Date:   Mon Oct 24 15:37:48 2016 +1000
Quoted 8 lines

drm/drivers: add support for using the arch wc mapping API.

is causing a huge performance regression for the ast drm driver. In a text console, if I call "cat" on a large text file, it takes almost twice as much time to be displayed and scrolled completely.

Can you please check that the ast driver portion of that commit is both correct and complete?

And in the meantime, what bad will happen if we just revert the ast portion of that commit?

Thanks,

Signature · 2 lines
Jean Delvare
SUSE L3 Support
#2 lore raw
Quoted 16 lines · Jean Delvare
Quoted 14 lines · Jean Delvare

Hi David,

The following commit:

commit 7cf321d118a825c1541b43ca45294126fd474efa

Author: Dave Airlie
Date:   Mon Oct 24 15:37:48 2016 +1000

drm/drivers: add support for using the arch wc mapping API.

is causing a huge performance regression for the ast drm driver. In a text console, if I call "cat" on a large text file, it takes almost twice as much time to be displayed and scrolled completely.

Quoted 6 lines
Quoted 2 lines

Can you please check that the ast driver portion of that commit is both correct and complete?

And in the meantime, what bad will happen if we just revert the ast portion of that commit?

This seems likely to be a hw problem with PCI writes to the AST "GPU", since it's just some sort of RAM + ARM on the end of a PCIE bus, we've definitely seen possible issues in the past with write combining around some of the mga GPUs with some CPUs.

Have we seen the problem across a number of AST devices?

Dave.

#3 lore raw
Quoted 6 lines · David Airlie

This seems likely to be a hw problem with PCI writes to the AST "GPU", since it's just some sort of RAM + ARM on the end of a PCIE bus, we've definitely seen possible issues in the past with write combining around some of the mga GPUs with some CPUs.

Have we seen the problem across a number of AST devices?

The reports I received from customers were all on AST 2500 devices (on Supermicro X11DPi-N, Supermicro X11DPH-T and Asus WS C621 Sage).

I was able to reproduce "the problem" on my old Asus Z6NA-D6 which has either an AST 2050 device if I trust the board specifications on asus.com, or an AST 1100 if I trust /var/log/Xorg.0.log.

Note however that I am still not certain that the problem I am seeing is the same as what both customers reported. It is possible that we have 2 different issues.

Signature · 2 lines
Jean Delvare
SUSE L3 Support
#4 lore raw
In reply to David Airlie

Hi David,

Quoted 20 lines · David Airlie
Quoted 19 lines · Jean Delvare
Quoted 16 lines · Jean Delvare

Hi David,

The following commit:

commit 7cf321d118a825c1541b43ca45294126fd474efa

Author: Dave Airlie
Date:   Mon Oct 24 15:37:48 2016 +1000

drm/drivers: add support for using the arch wc mapping API.

is causing a huge performance regression for the ast drm driver. In a text console, if I call "cat" on a large text file, it takes almost twice as much time to be displayed and scrolled completely.

Can you please check that the ast driver portion of that commit is both correct and complete?

Quoted 7 lines
Quoted 2 lines

And in the meantime, what bad will happen if we just revert the ast portion of that commit?

This seems likely to be a hw problem with PCI writes to the AST "GPU", since it's just some sort of RAM + ARM on the end of a PCIE bus, we've definitely seen possible issues in the past with write combining around some of the mga GPUs with some CPUs.

Takashi asked me to compare the contents of /sys/kernel/debug/x86/pat_memtype_list before and after the commit above. Before the commit, we have:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000
write-combining @ 0xfb005000-0xfb584000

After the commit, we have:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000
uncached-minus @ 0xfb000000-0xfb800000
uncached-minus @ 0xfb005000-0xfb584000

The corresponding lines in /proc/iomem are:

f0000000-fed8ffff : PCI Bus 0000:00
  faf00000-fb7fffff : PCI Bus 0000:01
    fafe0000-faffffff : 0000:01:01.0
    fb000000-fb7fffff : 0000:01:01.0

Does it help? Is the change of type expected? Is it not a problem that one of the ranges is overlapping with 2 other ranges?

Signature · 2 lines
Jean Delvare
SUSE L3 Support
#5 lore raw
Quoted 40 lines · Jean Delvare

Hi David,

Quoted 20 lines · David Airlie
Quoted 19 lines · Jean Delvare
Quoted 16 lines · Jean Delvare

Hi David,

The following commit:

commit 7cf321d118a825c1541b43ca45294126fd474efa

Author: Dave Airlie
Date:   Mon Oct 24 15:37:48 2016 +1000

drm/drivers: add support for using the arch wc mapping API.

is causing a huge performance regression for the ast drm driver. In a text console, if I call "cat" on a large text file, it takes almost twice as much time to be displayed and scrolled completely.

Can you please check that the ast driver portion of that commit is both correct and complete?

Quoted 7 lines
Quoted 2 lines

And in the meantime, what bad will happen if we just revert the ast portion of that commit?

This seems likely to be a hw problem with PCI writes to the AST "GPU", since it's just some sort of RAM + ARM on the end of a PCIE bus, we've definitely seen possible issues in the past with write combining around some of the mga GPUs with some CPUs.

Takashi asked me to compare the contents of /sys/kernel/debug/x86/pat_memtype_list before and after the commit above. Before the commit, we have:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000
write-combining @ 0xfb005000-0xfb584000

Quoted 6 lines

After the commit, we have:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000
uncached-minus @ 0xfb000000-0xfb800000
uncached-minus @ 0xfb005000-0xfb584000

Just to be sure:
could you double-check whether you're checking the right order (i.e. not checking against the revert)? The change above looks illogical from what I can see from the commit...

thanks,

Takashi

Quoted 15 lines

The corresponding lines in /proc/iomem are:

f0000000-fed8ffff : PCI Bus 0000:00
  faf00000-fb7fffff : PCI Bus 0000:01
    fafe0000-faffffff : 0000:01:01.0
    fb000000-fb7fffff : 0000:01:01.0

Does it help? Is the change of type expected? Is it not a problem that one of the ranges is overlapping with 2 other ranges?

Signature · 2 lines
Jean Delvare
SUSE L3 Support
#6 lore raw
Quoted 17 lines · Takashi Iwai
Quoted 8 lines · Jean Delvare

Takashi asked me to compare the contents of /sys/kernel/debug/x86/pat_memtype_list before and after the commit above. Before the commit, we have:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000
write-combining @ 0xfb005000-0xfb584000

Quoted 6 lines

After the commit, we have:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000
uncached-minus @ 0xfb000000-0xfb800000
uncached-minus @ 0xfb005000-0xfb584000

Quoted 4 lines

Just to be sure:
could you double-check whether you're checking the right order (i.e. not checking against the revert)? The change above looks illogical from what I can see from the commit...

Yes, I double checked and can only confirm what I wrote above. However while checking I noticed another strange thing: the contents of pat_memtype_list depend on whether I'm reading the file from an xterm or from the text console (Ctrl+Alt+F2). So I summarize again:

Commit "drm/drivers: add support for using the arch wc mapping API" APPLIED, reading pat_memtype_list from the text console:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000
uncached-minus @ 0xfb000000-0xfb800000
uncached-minus @ 0xfb005000-0xfb584000

Commit "drm/drivers: add support for using the arch wc mapping API" APPLIED, reading pat_memtype_list from an xterm:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000
uncached-minus @ 0xfb000000-0xfb800000

Commit "drm/drivers: add support for using the arch wc mapping API" REVERTED, reading pat_memtype_list from the text console:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000
write-combining @ 0xfb005000-0xfb584000

Commit "drm/drivers: add support for using the arch wc mapping API" REVERTED, reading pat_memtype_list from an xterm:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000

If you don't find it logical, remember that if things were the way they are supposed to be, customers and myself would not be reporting this bug ;-)

Signature · 2 lines
Jean Delvare
SUSE L3 Support
#10 lore raw
In reply to Jean Delvare

Hi Jean

Am 12.11.18 um 15:36 schrieb Jean Delvare:

Quoted 48 lines

Hi David,

Quoted 20 lines · David Airlie
Quoted 19 lines · Jean Delvare
Quoted 16 lines · Jean Delvare

Hi David,

The following commit:

commit 7cf321d118a825c1541b43ca45294126fd474efa

Author: Dave Airlie
Date:   Mon Oct 24 15:37:48 2016 +1000

drm/drivers: add support for using the arch wc mapping API.

is causing a huge performance regression for the ast drm driver. In a text console, if I call "cat" on a large text file, it takes almost twice as much time to be displayed and scrolled completely.

Can you please check that the ast driver portion of that commit is both correct and complete?

Quoted 7 lines
Quoted 2 lines

And in the meantime, what bad will happen if we just revert the ast portion of that commit?

This seems likely to be a hw problem with PCI writes to the AST "GPU", since it's just some sort of RAM + ARM on the end of a PCIE bus, we've definitely seen possible issues in the past with write combining around some of the mga GPUs with some CPUs.

Takashi asked me to compare the contents of /sys/kernel/debug/x86/pat_memtype_list before and after the commit above. Before the commit, we have:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000
write-combining @ 0xfb005000-0xfb584000

After the commit, we have:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000
uncached-minus @ 0xfb000000-0xfb800000
uncached-minus @ 0xfb005000-0xfb584000

The corresponding lines in /proc/iomem are:

Quoted 8 lines

f0000000-fed8ffff : PCI Bus 0000:00
  faf00000-fb7fffff : PCI Bus 0000:01
    fafe0000-faffffff : 0000:01:01.0
    fb000000-fb7fffff : 0000:01:01.0

Does it help? Is the change of type expected? Is it not a problem that one of the ranges is overlapping with 2 other ranges?

I debugged this problem with an AST 2000. What happens is that the vesafb driver attaches first to the graphics device. It disables write-combining and falls back to uncached access; unless the kernel has been booted with video=vesafb:mtrr:3. Then it will set-up WC correctly. The respective code is at [1].

Best regards
Thomas

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/video/fbdev/vesafb.c?h=v4.20-rc2#n415

Signature · 6 lines
Thomas Zimmermann
Graphics Driver Developer
SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nürnberg
Tel: +49-911-74053-0; Fax: +49-911-7417755;  https://www.suse.com/
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
Graham Norton, HRB 21284 (AG Nürnberg)
#11 lore raw
In reply to Jean Delvare

Sir,
We found the performance will be bad on desktop environment with OpenSUSE15 UEFI installation if remove "arch_io_reserve_memtype_wc".

Regards,

Y.C. Chen

-----Original Message-----
From: dri-devel [mailto:[email protected]] On Behalf Of Jean Delvare
Sent: Thursday, November 08, 2018 8:05 PM
To: Dave Airlie <[email protected]>
Cc: Christian König <[email protected]>; [email protected]; [email protected]
Subject: Re: Performance regression in ast drm driver

Quoted 8 lines · Jean Delvare

Hi David,

The following commit:

commit 7cf321d118a825c1541b43ca45294126fd474efa

Author: Dave Airlie
Date:   Mon Oct 24 15:37:48 2016 +1000
Quoted 8 lines

drm/drivers: add support for using the arch wc mapping API.

is causing a huge performance regression for the ast drm driver. In a text console, if I call "cat" on a large text file, it takes almost twice as much time to be displayed and scrolled completely.

Can you please check that the ast driver portion of that commit is both correct and complete?

And in the meantime, what bad will happen if we just revert the ast portion of that commit?

Thanks,
--
Jean Delvare
SUSE L3 Support
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#7 lore raw
In reply to Jean Delvare

Jean,

ast doesn't remove the vesafb's framebuffer before attaching to the device. I have a patch at [1]. If you have a way of testing it, I'd appreciate.

Best regards
Thomas

[1] https://bugzilla.suse.com/show_bug.cgi?id=1112963

Am 12.11.18 um 17:41 schrieb Jean Delvare:

Quoted 60 lines · Takashi Iwai
Quoted 17 lines
Quoted 8 lines · Jean Delvare

Takashi asked me to compare the contents of /sys/kernel/debug/x86/pat_memtype_list before and after the commit above. Before the commit, we have:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000
write-combining @ 0xfb005000-0xfb584000

Quoted 6 lines

After the commit, we have:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000
uncached-minus @ 0xfb000000-0xfb800000
uncached-minus @ 0xfb005000-0xfb584000

Quoted 4 lines

Just to be sure:
could you double-check whether you're checking the right order (i.e. not checking against the revert)? The change above looks illogical from what I can see from the commit...

Yes, I double checked and can only confirm what I wrote above. However while checking I noticed another strange thing: the contents of pat_memtype_list depend on whether I'm reading the file from an xterm or from the text console (Ctrl+Alt+F2). So I summarize again:

Commit "drm/drivers: add support for using the arch wc mapping API" APPLIED, reading pat_memtype_list from the text console:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000
uncached-minus @ 0xfb000000-0xfb800000
uncached-minus @ 0xfb005000-0xfb584000

Commit "drm/drivers: add support for using the arch wc mapping API" APPLIED, reading pat_memtype_list from an xterm:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000
uncached-minus @ 0xfb000000-0xfb800000

Commit "drm/drivers: add support for using the arch wc mapping API" REVERTED, reading pat_memtype_list from the text console:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000
write-combining @ 0xfb005000-0xfb584000

Commit "drm/drivers: add support for using the arch wc mapping API" REVERTED, reading pat_memtype_list from an xterm:

uncached-minus @ 0xfafe0000-0xfb000000
uncached-minus @ 0xfb000000-0xfb005000

If you don't find it logical, remember that if things were the way they are supposed to be, customers and myself would not be reporting this bug ;-)

Signature · 6 lines
Thomas Zimmermann
Graphics Driver Developer
SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nürnberg
Tel: +49-911-74053-0; Fax: +49-911-7417755;  https://www.suse.com/
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
Graham Norton, HRB 21284 (AG Nürnberg)

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel

#8 lore raw

Hi Thomas,

Quoted 5 lines · Thomas Zimmermann

ast doesn't remove the vesafb's framebuffer before attaching to the device. I have a patch at [1]. If you have a way of testing it, I'd appreciate.

[1] https://bugzilla.suse.com/show_bug.cgi?id=1112963

Thank you very much for looking into this bug. I tested the patch above and yes, it solves my console performance problem. The performance with your patch applied is roughly the same as with commit "drm/drivers: add support for using the arch wc mapping API" reverted.

Signature · 2 lines
Jean Delvare
SUSE L3 Support
#9 lore raw

Hi

Am 13.11.18 um 13:08 schrieb Jean Delvare:

Quoted 7 lines

Hi Thomas,

Quoted 4 lines · Thomas Zimmermann

ast doesn't remove the vesafb's framebuffer before attaching to the device. I have a patch at [1]. If you have a way of testing it, I'd appreciate.

Quoted 7 lines
Quoted 1 line

[1] https://bugzilla.suse.com/show_bug.cgi?id=1112963

Thank you very much for looking into this bug. I tested the patch above and yes, it solves my console performance problem. The performance with your patch applied is roughly the same as with commit "drm/drivers: add support for using the arch wc mapping API" reverted.

Great. I'll add it to SLE and also send it upstream soonish. Let's see what Edward reports. Thanks for building a kernel for him.

Best regards
Thomas

Signature · 6 lines
Thomas Zimmermann
Graphics Driver Developer
SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nürnberg
Tel: +49-911-74053-0; Fax: +49-911-7417755;  https://www.suse.com/
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
Graham Norton, HRB 21284 (AG Nürnberg)

_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel