Skip to contentSkip to conversation

drm/fbdev-generic: Fix framebuffer on big endian devices

PATCH Applied Thomas Huth 4 participants 9 messages Jun 27, 2024 – Jul 3, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/fbdev-generic: Fix framebuffer on big endian devices

Starting with kernel 6.7, the framebuffer text console is not working anymore with the virtio-gpu device on s390x hosts. Such big endian fb devices are usinga different pixel ordering than little endian devices, e.g. DRM_FORMAT_BGRX8888 instead of DRM_FORMAT_XRGB8888.

This used to work fine as long as drm_client_buffer_addfb() was still calling drm_mode_addfb() which called drm_driver_legacy_fb_format() internally to get the right format. But drm_client_buffer_addfb() has recently been reworked to call drm_mode_addfb2() instead with the format value that has been passed to it as a parameter (see commit 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()").

That format parameter is determined in drm_fbdev_generic_helper_fb_probe() via the drm_mode_legacy_fb_format() function - which only generates formats suitable for little endian devices. So to fix this issue switch to drm_driver_legacy_fb_format() here instead to take the device endianness into consideration.

Fixes 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()")
---
 drivers/gpu/drm/drm_fbdev_generic.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/drm_fbdev_generic.c +2 -1
index 97e579c33d84..1e200d815e1a 100644
@@ -84,7 +84,8 @@ static int drm_fbdev_generic_helper_fb_probe(struct drm_fb_helper *fb_helper,
 		    sizes->surface_width, sizes->surface_height,
 		    sizes->surface_bpp);
 
-	format = drm_mode_legacy_fb_format(sizes->surface_bpp, sizes->surface_depth);
+	format = drm_driver_legacy_fb_format(dev, sizes->surface_bpp,
+					     sizes->surface_depth);
 	buffer = drm_client_framebuffer_create(client, sizes->surface_width,
 					       sizes->surface_height, format);
 	if (IS_ERR(buffer))
Signature · 1 line
2.45.2
Reviewed-by on v1 1/1
#1 lore raw

Thomas Huth <[email protected]> writes:

Hello Thomas,

Quoted 30 lines

Starting with kernel 6.7, the framebuffer text console is not working anymore with the virtio-gpu device on s390x hosts. Such big endian fb devices are usinga different pixel ordering than little endian devices, e.g. DRM_FORMAT_BGRX8888 instead of DRM_FORMAT_XRGB8888.

This used to work fine as long as drm_client_buffer_addfb() was still calling drm_mode_addfb() which called drm_driver_legacy_fb_format() internally to get the right format. But drm_client_buffer_addfb() has recently been reworked to call drm_mode_addfb2() instead with the format value that has been passed to it as a parameter (see commit 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()").

That format parameter is determined in drm_fbdev_generic_helper_fb_probe() via the drm_mode_legacy_fb_format() function - which only generates formats suitable for little endian devices. So to fix this issue switch to drm_driver_legacy_fb_format() here instead to take the device endianness into consideration.

Fixes 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()")
---
 drivers/gpu/drm/drm_fbdev_generic.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/drm_fbdev_generic.c +0 -0
index 97e579c33d84..1e200d815e1a 100644

drivers/gpu/drm/drm_fbdev_generic.c static int drm_fbdev_generic_helper_fb_probe(struct drm_fb_helper *fb_helper, · line 91

Quoted 9 lines
drivers/gpu/drm/drm_fbdev_generic.c +2 -1
@@ -84,7 +84,8 @@ static int drm_fbdev_generic_helper_fb_probe(struct drm_fb_helper *fb_helper,
 		    sizes->surface_width, sizes->surface_height,
 		    sizes->surface_bpp);
 
-	format = drm_mode_legacy_fb_format(sizes->surface_bpp, sizes->surface_depth);
+	format = drm_driver_legacy_fb_format(dev, sizes->surface_bpp,
+					     sizes->surface_depth);
 	buffer = drm_client_framebuffer_create(client, sizes->surface_width,
 					       sizes->surface_height, format);
 	if (IS_ERR(buffer))

Indeed. Thanks a lot for the detailed explanation and the patch. Your fix makes sense to me.

Signature · 5 lines
Best regards,

Javier Martinez Canillas
Core Platforms
Red Hat
maintainer Acked-by on v1 1/1
#2 lore raw
In reply to Thomas Huth

Hi

Am 27.06.24 um 19:35 schrieb Thomas Huth:

Quoted 18 lines

Starting with kernel 6.7, the framebuffer text console is not working anymore with the virtio-gpu device on s390x hosts. Such big endian fb devices are usinga different pixel ordering than little endian devices, e.g. DRM_FORMAT_BGRX8888 instead of DRM_FORMAT_XRGB8888.

This used to work fine as long as drm_client_buffer_addfb() was still calling drm_mode_addfb() which called drm_driver_legacy_fb_format() internally to get the right format. But drm_client_buffer_addfb() has recently been reworked to call drm_mode_addfb2() instead with the format value that has been passed to it as a parameter (see commit 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()").

That format parameter is determined in drm_fbdev_generic_helper_fb_probe() via the drm_mode_legacy_fb_format() function - which only generates formats suitable for little endian devices. So to fix this issue switch to drm_driver_legacy_fb_format() here instead to take the device endianness into consideration.

Quoted 3 lines
Fixes 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()")
Quoted 2 lines
---
  drivers/gpu/drm/drm_fbdev_generic.c | 3 ++-

This file is now called drm_fbdev_ttm.c in drm-misc-next. And a similar patch might be necessary for drm_fbdev_dma.c. The code in drm_fbdev_shmem.c apparently has it already.

Best regards
Thomas

Quoted 16 lines
  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/drm_fbdev_generic.c +2 -1
index 97e579c33d84..1e200d815e1a 100644
@@ -84,7 +84,8 @@ static int drm_fbdev_generic_helper_fb_probe(struct drm_fb_helper *fb_helper,
  		    sizes->surface_width, sizes->surface_height,
  		    sizes->surface_bpp);
 
-	format = drm_mode_legacy_fb_format(sizes->surface_bpp, sizes->surface_depth);
+	format = drm_driver_legacy_fb_format(dev, sizes->surface_bpp,
+					     sizes->surface_depth);
  	buffer = drm_client_framebuffer_create(client, sizes->surface_width,
  					       sizes->surface_height, format);
  	if (IS_ERR(buffer))
Signature · 7 lines
--
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Frankenstrasse 146, 90461 Nuernberg, Germany
GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman
HRB 36809 (AG Nuernberg)
#3 lore raw

Hi Thomas,

Quoted 26 lines · Thomas Zimmermann

Am 27.06.24 um 19:35 schrieb Thomas Huth:

Quoted 18 lines

Starting with kernel 6.7, the framebuffer text console is not working anymore with the virtio-gpu device on s390x hosts. Such big endian fb devices are usinga different pixel ordering than little endian devices, e.g. DRM_FORMAT_BGRX8888 instead of DRM_FORMAT_XRGB8888.

This used to work fine as long as drm_client_buffer_addfb() was still calling drm_mode_addfb() which called drm_driver_legacy_fb_format() internally to get the right format. But drm_client_buffer_addfb() has recently been reworked to call drm_mode_addfb2() instead with the format value that has been passed to it as a parameter (see commit 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()").

That format parameter is determined in drm_fbdev_generic_helper_fb_probe() via the drm_mode_legacy_fb_format() function - which only generates formats suitable for little endian devices. So to fix this issue switch to drm_driver_legacy_fb_format() here instead to take the device endianness into consideration.

Quoted 3 lines
Fixes 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()")
Quoted 6 lines
Quoted 2 lines
---
  drivers/gpu/drm/drm_fbdev_generic.c | 3 ++-

This file is now called drm_fbdev_ttm.c in drm-misc-next. And a similar patch might be necessary for drm_fbdev_dma.c. The code in drm_fbdev_shmem.c apparently has it already.

We are getting too many copies of this logic... (yup, had to fix them all up in my WIP support for R1 ;-)

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#4 lore raw
In reply to Thomas Zimmermann
Quoted 25 lines · Thomas Zimmermann

Hi

Am 27.06.24 um 19:35 schrieb Thomas Huth:

Quoted 18 lines

Starting with kernel 6.7, the framebuffer text console is not working anymore with the virtio-gpu device on s390x hosts. Such big endian fb devices are usinga different pixel ordering than little endian devices, e.g. DRM_FORMAT_BGRX8888 instead of DRM_FORMAT_XRGB8888.

This used to work fine as long as drm_client_buffer_addfb() was still calling drm_mode_addfb() which called drm_driver_legacy_fb_format() internally to get the right format. But drm_client_buffer_addfb() has recently been reworked to call drm_mode_addfb2() instead with the format value that has been passed to it as a parameter (see commit 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()").

That format parameter is determined in drm_fbdev_generic_helper_fb_probe() via the drm_mode_legacy_fb_format() function - which only generates formats suitable for little endian devices. So to fix this issue switch to drm_driver_legacy_fb_format() here instead to take the device endianness into consideration.

Quoted 3 lines
Fixes 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()")
Quoted 7 lines
Quoted 2 lines

---
  drivers/gpu/drm/drm_fbdev_generic.c | 3 ++-

This file is now called drm_fbdev_ttm.c in drm-misc-next.

Oh, ok, shall I send a v2 that is adjusted to that change, or can it be fixed while applying my patch?

Quoted 1 line

And a similar patch might be necessary for drm_fbdev_dma.c.

Looks similar, indeed. Shall I send a patch for that one, too? ... I currently don't have a setup for testing that, though...

Thomas

Quoted 23 lines

The code in drm_fbdev_shmem.c apparently has it already.

Best regards
Thomas

Quoted 17 lines
  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/drm_fbdev_generic.c +2 -1
index 97e579c33d84..1e200d815e1a 100644
@@ -84,7 +84,8 @@ static int drm_fbdev_generic_helper_fb_probe(struct 
 drm_fb_helper *fb_helper,
               sizes->surface_width, sizes->surface_height,
               sizes->surface_bpp);
-    format = drm_mode_legacy_fb_format(sizes->surface_bpp, 
 sizes->surface_depth);
+    format = drm_driver_legacy_fb_format(dev, sizes->surface_bpp,
+                         sizes->surface_depth);
       buffer = drm_client_framebuffer_create(client, sizes->surface_width,
                              sizes->surface_height, format);

      if (IS_ERR(buffer))

#5 lore raw

Hi Thomas,

Quoted 29 lines · Thomas Huth
Quoted 28 lines · Thomas Zimmermann

Am 27.06.24 um 19:35 schrieb Thomas Huth:

Quoted 19 lines

Starting with kernel 6.7, the framebuffer text console is not working anymore with the virtio-gpu device on s390x hosts. Such big endian fb devices are usinga different pixel ordering than little endian devices, e.g. DRM_FORMAT_BGRX8888 instead of DRM_FORMAT_XRGB8888.

This used to work fine as long as drm_client_buffer_addfb() was still calling drm_mode_addfb() which called drm_driver_legacy_fb_format() internally to get the right format. But drm_client_buffer_addfb() has recently been reworked to call drm_mode_addfb2() instead with the format value that has been passed to it as a parameter (see commit 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()").

That format parameter is determined in drm_fbdev_generic_helper_fb_probe() via the drm_mode_legacy_fb_format() function - which only generates formats suitable for little endian devices. So to fix this issue switch to drm_driver_legacy_fb_format() here instead to take the device endianness into consideration.

Quoted 4 lines
Fixes 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to

drm_mode_addfb2()")

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
---
  drivers/gpu/drm/drm_fbdev_generic.c | 3 ++-

This file is now called drm_fbdev_ttm.c in drm-misc-next.

Oh, ok, shall I send a v2 that is adjusted to that change, or can it be fixed while applying my patch?

As this is a regression in mainline, which needs to be backported, too, it's best to apply your fix to v6.10-rc6, which does not have drm_fbdev_ttm.c yet.

Quoted 4 lines
Quoted 1 line

And a similar patch might be necessary for drm_fbdev_dma.c.

Looks similar, indeed. Shall I send a patch for that one, too? ... I currently don't have a setup for testing that, though...

Obviously these need to be fixed, too.

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#6 lore raw

Hi

Am 01.07.24 um 10:54 schrieb Geert Uytterhoeven:

Quoted 31 lines

Hi Thomas,

Quoted 28 lines · Thomas Huth
Quoted 27 lines · Thomas Zimmermann

Am 27.06.24 um 19:35 schrieb Thomas Huth:

Quoted 19 lines

Starting with kernel 6.7, the framebuffer text console is not working anymore with the virtio-gpu device on s390x hosts. Such big endian fb devices are usinga different pixel ordering than little endian devices, e.g. DRM_FORMAT_BGRX8888 instead of DRM_FORMAT_XRGB8888.

This used to work fine as long as drm_client_buffer_addfb() was still calling drm_mode_addfb() which called drm_driver_legacy_fb_format() internally to get the right format. But drm_client_buffer_addfb() has recently been reworked to call drm_mode_addfb2() instead with the format value that has been passed to it as a parameter (see commit 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()").

That format parameter is determined in drm_fbdev_generic_helper_fb_probe() via the drm_mode_legacy_fb_format() function - which only generates formats suitable for little endian devices. So to fix this issue switch to drm_driver_legacy_fb_format() here instead to take the device endianness into consideration.

Quoted 4 lines
Fixes 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to

drm_mode_addfb2()")

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
---
   drivers/gpu/drm/drm_fbdev_generic.c | 3 ++-

This file is now called drm_fbdev_ttm.c in drm-misc-next.

Oh, ok, shall I send a v2 that is adjusted to that change, or can it be fixed while applying my patch?

As this is a regression in mainline, which needs to be backported, too, it's best to apply your fix to v6.10-rc6, which does not have drm_fbdev_ttm.c yet.

I guess we can put the current fix into drm-misc-fixes.

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

And a similar patch might be necessary for drm_fbdev_dma.c.

Looks similar, indeed. Shall I send a patch for that one, too? ... I currently don't have a setup for testing that, though...

It's up to you. You'd need drm-misc-next from https://gitlab.freedesktop.org/drm/misc/kernel

Best regards
Thomas

Quoted 6 lines

Obviously these need to be fixed, too.

Gr{oetje,eeting}s,

Geert

Signature · 7 lines
--
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Frankenstrasse 146, 90461 Nuernberg, Germany
GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman
HRB 36809 (AG Nuernberg)
Reviewed-by Tested-by on v1 1/1
#7 lore raw
In reply to Thomas Huth

Hi Thomas,

CC Christian

Quoted 18 lines · Thomas Huth

Starting with kernel 6.7, the framebuffer text console is not working anymore with the virtio-gpu device on s390x hosts. Such big endian fb devices are usinga different pixel ordering than little endian devices, e.g. DRM_FORMAT_BGRX8888 instead of DRM_FORMAT_XRGB8888.

This used to work fine as long as drm_client_buffer_addfb() was still calling drm_mode_addfb() which called drm_driver_legacy_fb_format() internally to get the right format. But drm_client_buffer_addfb() has recently been reworked to call drm_mode_addfb2() instead with the format value that has been passed to it as a parameter (see commit 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()").

That format parameter is determined in drm_fbdev_generic_helper_fb_probe() via the drm_mode_legacy_fb_format() function - which only generates formats suitable for little endian devices. So to fix this issue switch to drm_driver_legacy_fb_format() here instead to take the device endianness into consideration.

Quoted 3 lines
Fixes 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()")

works fine on m68k-virt, so:

Christian had reported a similar issue before[1].
I submitted a different solution fixing virtio[2] instead, but that caused issues with virtio-mouse-pci cursor...

drivers/gpu/drm/drm_fbdev_generic.c static int drm_fbdev_generic_helper_fb_probe(struct drm_fb_helper *fb_helper, · line 91

Quoted 12 lines
drivers/gpu/drm/drm_fbdev_generic.c +2 -1
@@ -84,7 +84,8 @@ static int drm_fbdev_generic_helper_fb_probe(struct drm_fb_helper *fb_helper,
                    sizes->surface_width, sizes->surface_height,
                    sizes->surface_bpp);
 
-       format = drm_mode_legacy_fb_format(sizes->surface_bpp, sizes->surface_depth);
+       format = drm_driver_legacy_fb_format(dev, sizes->surface_bpp,
+                                            sizes->surface_depth);
        buffer = drm_client_framebuffer_create(client, sizes->surface_width,
                                               sizes->surface_height, format);
        if (IS_ERR(buffer))

[1] https://lore.kernel.org/[email protected]/
[2] "[PATCH v2] drm/virtio: Add suppport for non-native buffer formats"

    https://lore.kernel.org/47a81d2e0e47b1715718779b6978a8b595cc7c5d.1700140609.git.geert@linux-m68k.org

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
maintainer Applied
#8 lore raw
In reply to Thomas Huth

Merged into drm-misc-fixes.

Am 27.06.24 um 19:35 schrieb Thomas Huth:

Quoted 39 lines

Starting with kernel 6.7, the framebuffer text console is not working anymore with the virtio-gpu device on s390x hosts. Such big endian fb devices are usinga different pixel ordering than little endian devices, e.g. DRM_FORMAT_BGRX8888 instead of DRM_FORMAT_XRGB8888.

This used to work fine as long as drm_client_buffer_addfb() was still calling drm_mode_addfb() which called drm_driver_legacy_fb_format() internally to get the right format. But drm_client_buffer_addfb() has recently been reworked to call drm_mode_addfb2() instead with the format value that has been passed to it as a parameter (see commit 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()").

That format parameter is determined in drm_fbdev_generic_helper_fb_probe() via the drm_mode_legacy_fb_format() function - which only generates formats suitable for little endian devices. So to fix this issue switch to drm_driver_legacy_fb_format() here instead to take the device endianness into consideration.

Fixes 6ae2ff23aa43 ("drm/client: Convert drm_client_buffer_addfb() to drm_mode_addfb2()")
---
  drivers/gpu/drm/drm_fbdev_generic.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/drm_fbdev_generic.c +2 -1
index 97e579c33d84..1e200d815e1a 100644
@@ -84,7 +84,8 @@ static int drm_fbdev_generic_helper_fb_probe(struct drm_fb_helper *fb_helper,
  		    sizes->surface_width, sizes->surface_height,
  		    sizes->surface_bpp);
 
-	format = drm_mode_legacy_fb_format(sizes->surface_bpp, sizes->surface_depth);
+	format = drm_driver_legacy_fb_format(dev, sizes->surface_bpp,
+					     sizes->surface_depth);
  	buffer = drm_client_framebuffer_create(client, sizes->surface_width,
  					       sizes->surface_height, format);
  	if (IS_ERR(buffer))
Signature · 7 lines
--
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Frankenstrasse 146, 90461 Nuernberg, Germany
GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman
HRB 36809 (AG Nuernberg)