Skip to contentSkip to conversation

`io{re,un}map()` build error in s390 under `!CONFIG_HAS_IOMEM`

DISCUSSION Miguel Ojeda 8 participants 20 messages Aug 3 – Aug 12

Subsystems (4)
lore
Share & tools
lore

Miguel Ojeda

author maintainer
#0 lore raw

Hi all,

s390 does not provide ioremap()/iounmap() under !CONFIG_HAS_IOMEM:

In file included from rust/helpers/helpers.c:68:
rust/helpers/io.c:8:9: error: call to undeclared function 'ioremap'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
8 | return ioremap(offset, size);
| ^
rust/helpers/io.c:8:9: error: incompatible integer to pointer conversion returning 'int' from a function with result type 'void *' [-Wint-conversion]
8 | return ioremap(offset, size);
| ^~~~~~~~~~~~~~~~~~~~~
rust/helpers/io.c:19:2: error: call to undeclared function 'iounmap'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
19 | iounmap(addr);
| ^
./arch/s390/include/asm/io.h:31:17: note: expanded from macro 'iounmap'
31 | #define iounmap iounmap
| ^
rust/helpers/io.c:19:2: note: did you mean 'kunmap'?
./arch/s390/include/asm/io.h:31:17: note: expanded from macro 'iounmap'
31 | #define iounmap iounmap
| ^
./include/linux/highmem.h:46:20: note: 'kunmap' declared here
46 | static inline void kunmap(const struct page *page);
| ^

so e.g. an allnoconfig with Rust fails.

I assume we will need to #ifdef the helpers, plus the users; or stub at some layer to avoid changing callers.

I hope that helps & thanks!

Cheers,
Miguel

maintainer
#1 lore raw
Quoted 7 lines · Miguel Ojeda

so e.g. an allnoconfig with Rust fails.

I assume we will need to #ifdef the helpers, plus the users; or stub at some layer to avoid changing callers.

I hope that helps & thanks!

I think in practice you'll find that all sensible configurations will have PCI enabled since that was introduced in the 2012 generation (zEC12), while the oldest supported machines in the kernel these days are from 2008 (z10) and Red Hat Enterprise Linux already needs a z14 from 2017.

In theory you should be able to use rust code without PCI MMIO support, but I can't see any practical downsides to making rust 'depends on HAS_MMIO' to avoid having to add those #ifdef.

Note that we already have 'config COMPILE_TEST depends on HAS_MMIO' to work around similar problems with random drivers expecting MMIO support.

Arnd

maintainer
#2 lore raw
Quoted 3 lines · Arnd Bergmann

In theory you should be able to use rust code without PCI MMIO support, but I can't see any practical downsides to making rust 'depends on HAS_MMIO' to avoid having to add those #ifdef.

I think the implications should be minor without making Rust depend on CONFIG_HAS_IOMEM.

I just sent out a fix [1]; the only annoying part is [2], but we should change those doc-tests anyway. For the one in rust/kernel/io.rs we already did in driver-core-next.

[1] https://lore.kernel.org/driver-core/[email protected]/
[2] https://lore.kernel.org/driver-core/[email protected]/

maintainer
#3 lore raw
Quoted 8 lines · Danilo Krummrich
Quoted 3 lines · Arnd Bergmann

In theory you should be able to use rust code without PCI MMIO support, but I can't see any practical downsides to making rust 'depends on HAS_MMIO' to avoid having to add those #ifdef.

I think the implications should be minor without making Rust depend on CONFIG_HAS_IOMEM.

Quoted 6 lines

I just sent out a fix [1]; the only annoying part is [2], but we should change those doc-tests anyway. For the one in rust/kernel/io.rs we already did in driver-core-next.

[1] https://lore.kernel.org/driver-core/[email protected]/
[2] https://lore.kernel.org/driver-core/[email protected]/

I'm wondering if it would make sense to make HAS_IOMEM always available on s390, even though it doesn't make too much sense without PCI. But at least it would make s390 again a bit less special.

Niklas, Gerd?

maintainer
#4 lore raw
Quoted 6 lines · Heiko Carstens
Quoted 5 lines · Danilo Krummrich
Quoted 3 lines · Arnd Bergmann

In theory you should be able to use rust code without PCI MMIO support, but I can't see any practical downsides to making rust 'depends on HAS_MMIO' to avoid having to add those #ifdef.

Quoted 8 lines
Quoted 8 lines

I think the implications should be minor without making Rust depend on CONFIG_HAS_IOMEM.

I just sent out a fix [1]; the only annoying part is [2], but we should change those doc-tests anyway. For the one in rust/kernel/io.rs we already did in driver-core-next.

[1] https://lore.kernel.org/driver-core/[email protected]/

This looks like you still provide the rust version of ioremap(), turning what is supposed to be a link failure into a runtime error.

Quoted 3 lines

I'm wondering if it would make sense to make HAS_IOMEM always available on s390, even though it doesn't make too much sense without PCI. But at least it would make s390 again a bit less special.

I see that with CONFIG_PCI=y, s390 already falls back to generic_ioremap_prot() and just maps any phys_addr_t into the page table as PAGE_KERNEL, regardless of whether this is an MMIO address or not.

The simple change below would just extend that behavior to !PCI and make that consistent with CONFIG_PCI=y on machines without actual PCI hardware. Of course any code that might rely on this is now a bug that likely never gets caught at build time.

This still relies on implementing the __raw_* helpers as nop to have the same behavior as the PCI=y version, as the generic version would just end up dereferencing the invalid pointers.

Arnd

arch/s390/Kconfig +1 -4
index 84404e6778d5..806782acf6f6 100644
@@ -173,7 +173,7 @@ config S390
 	select GENERIC_ENTRY
 	select GENERIC_GETTIMEOFDAY
 	select GENERIC_SMP_IDLE_THREAD
-	select GENERIC_IOREMAP if PCI
+	select GENERIC_IOREMAP
 	select GLOB
 	select HAVE_ALIGNED_STRUCT_PAGE
 	select HAVE_ARCH_AUDITSYSCALL
@@ -756,9 +756,6 @@ config PCI_NR_FUNCTIONS
 
 endif # PCI
 
-config HAS_IOMEM
-	def_bool PCI
-
 config CHSC_SCH
 	def_tristate m
 	prompt "Support for CHSC subchannels"
arch/s390/include/asm/io.h +46 -17

82 lines not on this page · raw message

init/Kconfig +0 -1
index 5230d4879b1c..2a4e3ecbfddd 100644
@@ -237,7 +237,6 @@ config INIT_ENV_ARG_LIMIT
 
 config COMPILE_TEST
 	bool "Compile also drivers which will not load"
-	depends on HAS_IOMEM
 	help
 	  Some drivers can be compiled on a different platform than they are
 	  intended to be run on. Despite they cannot be loaded there (or even
maintainer
#5 lore raw
Quoted 14 lines · Arnd Bergmann
Quoted 13 lines · Heiko Carstens
Quoted 12 lines · Danilo Krummrich
Quoted 3 lines · Arnd Bergmann

In theory you should be able to use rust code without PCI MMIO support, but I can't see any practical downsides to making rust 'depends on HAS_MMIO' to avoid having to add those #ifdef.

I think the implications should be minor without making Rust depend on CONFIG_HAS_IOMEM.

I just sent out a fix [1]; the only annoying part is [2], but we should change those doc-tests anyway. For the one in rust/kernel/io.rs we already did in driver-core-next.

Quoted 5 lines
Quoted 1 line
Quoted 1 line

[1] https://lore.kernel.org/driver-core/[email protected]/

This looks like you still provide the rust version of ioremap(), turning what is supposed to be a link failure into a runtime error.

Which is the standard for many core APIs, such as [1]. However, I do agree that in this case the correct fix would be to have all architectures provide the stubs rather than the Rust code.

However, there's already a precedent for this in the kernel, e.g. in [2]. Of course, it would be better to clean this up, but depending on whether there's more architectures having this issue (I didn't check) that's separate from a fix.

[1] https://elixir.bootlin.com/linux/v7.1.5/source/include/linux/regmap.h#L1809
[2] https://elixir.bootlin.com/linux/v7.1.5/source/include/linux/device/devres.h#L115

Quoted 9 lines
Quoted 3 lines

I'm wondering if it would make sense to make HAS_IOMEM always available on s390, even though it doesn't make too much sense without PCI. But at least it would make s390 again a bit less special.

I see that with CONFIG_PCI=y, s390 already falls back to generic_ioremap_prot() and just maps any phys_addr_t into the page table as PAGE_KERNEL, regardless of whether this is an MMIO address or not.

Quoted 8 lines

The simple change below would just extend that behavior to !PCI and make that consistent with CONFIG_PCI=y on machines without actual PCI hardware. Of course any code that might rely on this is now a bug that likely never gets caught at build time.

This still relies on implementing the __raw_* helpers as nop to have the same behavior as the PCI=y version, as the generic version would just end up dereferencing the invalid pointers.

As mentioned, I didn't check, but if this is the only architecture causing those issues that'd be the better fix of course.

However, IIUC, your patch below would make ioremap() and friends silenty succeed and only the accessors would prevent undefined behavior?

In this case I still think ioremap() should just fail.

maintainer
#6 lore raw
Quoted 9 lines · Danilo Krummrich
Quoted 8 lines · Arnd Bergmann
Quoted 6 lines · Heiko Carstens
Quoted 6 lines

I just sent out a fix [1]; the only annoying part is [2], but we should change those doc-tests anyway. For the one in rust/kernel/io.rs we already did in driver-core-next.

[1] https://lore.kernel.org/driver-core/[email protected]/

Quoted 7 lines
Quoted 3 lines

This looks like you still provide the rust version of ioremap(), turning what is supposed to be a link failure into a runtime error.

Which is the standard for many core APIs, such as [1]. However, I do agree that in this case the correct fix would be to have all architectures provide the stubs rather than the Rust code.

We have both types of interfaces in the kernel. For HAS_IOMEM and HAS_IOPORT, the link failure is intentional, as it helps identify drivers that need a Kconfig dependency and are either unusable or potentially harmful if loaded without this.

Having empty stubs only really makes sense for things like LED support where a driver calling the interfaces can continue to work correctly when the interface is compile-time disabled.

Quoted 4 lines

However, there's already a precedent for this in the kernel, e.g. in [2]. Of course, it would be better to clean this up, but depending on whether there's more architectures having this issue (I didn't check) that's separate from a fix.

arch/um is the only other one that does not always enable HAS_IOMEM, though most m68k targets don't have any support for ISA/PCI style MMIO or PIO and probably should not enable it in theory.

Quoted 2 lines

[2]
https://elixir.bootlin.com/linux/v7.1.5/source/include/linux/device/devres.h#L115

Right, we are definitely already inconsistent here.

Quoted 9 lines
Quoted 8 lines

The simple change below would just extend that behavior to !PCI and make that consistent with CONFIG_PCI=y on machines without actual PCI hardware. Of course any code that might rely on this is now a bug that likely never gets caught at build time.

This still relies on implementing the __raw_* helpers as nop to have the same behavior as the PCI=y version, as the generic version would just end up dereferencing the invalid pointers.

Quoted 7 lines

As mentioned, I didn't check, but if this is the only architecture causing those issues that'd be the better fix of course.

However, IIUC, your patch below would make ioremap() and friends silenty succeed and only the accessors would prevent undefined behavior?

In this case I still think ioremap() should just fail.

In that case, it would make sense to also change the CONFIG_PCI=y version to fail the same way when the address points outside of the PCI memory space range. The current version in arch/s390/pci/pci.c just falls back to generic_ioremap_prot(), which is what I would use here directly:

void __iomem *ioremap_prot(phys_addr_t phys_addr, size_t size,
pgprot_t prot)

{
        if (!static_branch_unlikely(&have_mio))
                return (void __iomem *)phys_addr;
        return generic_ioremap_prot(phys_addr, size, prot);
}

The two methods here (generic_ioremap_prot() and the cast) are machine specific to refer to two different ways that PCI devices can be accessed if present, but there is no case for PCI being unavailable altogether.

Arnd

reviewer
#7 lore raw
Quoted 18 lines · Arnd Bergmann
Quoted 9 lines · Danilo Krummrich
Quoted 8 lines · Arnd Bergmann
Quoted 6 lines · Heiko Carstens
Quoted 6 lines

I just sent out a fix [1]; the only annoying part is [2], but we should change those doc-tests anyway. For the one in rust/kernel/io.rs we already did in driver-core-next.

[1] https://lore.kernel.org/driver-core/[email protected]/

Quoted 7 lines
Quoted 3 lines

This looks like you still provide the rust version of ioremap(), turning what is supposed to be a link failure into a runtime error.

Which is the standard for many core APIs, such as [1]. However, I do agree that in this case the correct fix would be to have all architectures provide the stubs rather than the Rust code.

Quoted 8 lines

We have both types of interfaces in the kernel. For HAS_IOMEM and HAS_IOPORT, the link failure is intentional, as it helps identify drivers that need a Kconfig dependency and are either unusable or potentially harmful if loaded without this.

Having empty stubs only really makes sense for things like LED support where a driver calling the interfaces can continue to work correctly when the interface is compile-time disabled.

We can still have link failures if we always provide the signatures, just don't provide implementation?

Best,
Gary

Quoted 52 lines
Quoted 5 lines

However, there's already a precedent for this in the kernel, e.g. in [2]. Of course, it would be better to clean this up, but depending on whether there's more architectures having this issue (I didn't check) that's separate from a fix.

arch/um is the only other one that does not always enable HAS_IOMEM, though most m68k targets don't have any support for ISA/PCI style MMIO or PIO and probably should not enable it in theory.

Quoted 2 lines

[2]
https://elixir.bootlin.com/linux/v7.1.5/source/include/linux/device/devres.h#L115

Right, we are definitely already inconsistent here.

Quoted 9 lines
Quoted 8 lines

The simple change below would just extend that behavior to !PCI and make that consistent with CONFIG_PCI=y on machines without actual PCI hardware. Of course any code that might rely on this is now a bug that likely never gets caught at build time.

This still relies on implementing the __raw_* helpers as nop to have the same behavior as the PCI=y version, as the generic version would just end up dereferencing the invalid pointers.

Quoted 7 lines

As mentioned, I didn't check, but if this is the only architecture causing those issues that'd be the better fix of course.

However, IIUC, your patch below would make ioremap() and friends silenty succeed and only the accessors would prevent undefined behavior?

In this case I still think ioremap() should just fail.

In that case, it would make sense to also change the CONFIG_PCI=y version to fail the same way when the address points outside of the PCI memory space range. The current version in arch/s390/pci/pci.c just falls back to generic_ioremap_prot(), which is what I would use here directly:

void __iomem *ioremap_prot(phys_addr_t phys_addr, size_t size,
pgprot_t prot)

{
        if (!static_branch_unlikely(&have_mio))
                return (void __iomem *)phys_addr;
        return generic_ioremap_prot(phys_addr, size, prot);
}

The two methods here (generic_ioremap_prot() and the cast) are machine specific to refer to two different ways that PCI devices can be accessed if present, but there is no case for PCI being unavailable altogether.

Arnd

maintainer
#8 lore raw
In reply to Arnd Bergmann
Quoted 14 lines · Arnd Bergmann

In that case, it would make sense to also change the CONFIG_PCI=y version to fail the same way when the address points outside of the PCI memory space range. The current version in arch/s390/pci/pci.c just falls back to generic_ioremap_prot(), which is what I would use here directly:

void __iomem *ioremap_prot(phys_addr_t phys_addr, size_t size,
pgprot_t prot)

{
        if (!static_branch_unlikely(&have_mio))
                return (void __iomem *)phys_addr;
        return generic_ioremap_prot(phys_addr, size, prot);
}
Quoted 4 lines

The two methods here (generic_ioremap_prot() and the cast) are machine specific to refer to two different ways that PCI devices can be accessed if present, but there is no case for PCI being unavailable altogether.

I think we agree the correct solution is to fix things in s390. But the conversation seems to suggest that this rather results into merge window material, so I will go ahead and pull in the other fixes for -rc7.

Thanks,
Danilo

#9 lore raw
In reply to Arnd Bergmann
Quoted 3 lines · Arnd Bergmann

arch/um is the only other one that does not always enable HAS_IOMEM, though most m68k targets don't have any support for ISA/PCI style MMIO or PIO and probably should not enable it in theory.

While HAS_IOPORT is indeed a feature that has propagated from Intel x86
(or really 8080, etc.) to non-x86 systems through ISA/PCI what's been so special about ISA/PCI WRT HAS_IOMEM?

AFAICT HAS_IOMEM is all about just having peripherals located in the
CPU's regular memory addressing space. All the DEC MIPS platforms fall into this category, never seen PCI/ISA, some have no peripheral bus.

Maciej

maintainer
#10 lore raw
Quoted 6 lines · Maciej W. Rozycki
Quoted 3 lines · Arnd Bergmann

arch/um is the only other one that does not always enable HAS_IOMEM, though most m68k targets don't have any support for ISA/PCI style MMIO or PIO and probably should not enable it in theory.

Quoted 7 lines

While HAS_IOPORT is indeed a feature that has propagated from Intel x86
(or really 8080, etc.) to non-x86 systems through ISA/PCI what's been so special about ISA/PCI WRT HAS_IOMEM?

AFAICT HAS_IOMEM is all about just having peripherals located in the
CPU's regular memory addressing space. All the DEC MIPS platforms fall into this category, never seen PCI/ISA, some have no peripheral bus.

To me this is more about the driver model: On most architectures, you can use portable drivers that use IORESOURCE_MEM resources with ioremap() and the do readl()/writel() on those to access little-endian registers.

s390 and um only support this conceptually for PCI devices, and the HW implementation is different enough that the abstraction breaks down in some cases (like memremap()). m68k Amiga is similar to decstation I think, in that it has devices that are memory mapped, but they are always visible in the kernel address space and are accessed through their own abstraction instead of readl()/writel(). m68k q40 does have ISA devices with both port and memory accesses that can use portable drivers. I think other m68k platforms use a mix of ioremap()/readl()/writel() and their own lower-level accessors based on the driver, but none of those drivers are shared with another platform.

Arnd

#11 lore raw
Quoted 8 lines · Arnd Bergmann
Quoted 3 lines

AFAICT HAS_IOMEM is all about just having peripherals located in the
CPU's regular memory addressing space. All the DEC MIPS platforms fall into this category, never seen PCI/ISA, some have no peripheral bus.

To me this is more about the driver model: On most architectures, you can use portable drivers that use IORESOURCE_MEM resources with ioremap() and the do readl()/writel() on those to access little-endian registers.

Obviously I can't speak for the s390, but that's been my observation as
well -- it is a matter of coding style then rather than HAS_IOMEM being inherently PCI/ISA.

Quoted 11 lines

s390 and um only support this conceptually for PCI devices, and the HW implementation is different enough that the abstraction breaks down in some cases (like memremap()). m68k Amiga is similar to decstation I think, in that it has devices that are memory mapped, but they are always visible in the kernel address space and are accessed through their own abstraction instead of readl()/writel(). m68k q40 does have ISA devices with both port and memory accesses that can use portable drivers. I think other m68k platforms use a mix of ioremap()/readl()/writel() and their own lower-level accessors based on the driver, but none of those drivers are shared with another platform.

On DECstation we actually have driver code shared between TURBOchannel
peripheral bus and PCI (and possibly EISA) and it obviously uses the standard MMIO accessors consistently regardless of the bus binding.

Then there is code shared between TURBOchannel and platform stuff, also
with the system models that lack a peripheral bus too. There is leftover code using non-standard accessors still there, but that's really a matter of resources needed to clean this up.

For instance drivers/net/ethernet/amd/declance.c has now been high on my
priority list to clean up and properly convert from its current mess to a regular TURBOchannel/platform device. The only issue outstanding are MMIO barriers, which have never been properly sorted out in a generic way; cf. arch/mips/include/asm/io.h and the iobarrier_*() stuff there.

Maciej

#12 lore raw
In reply to Arnd Bergmann
Quoted 34 lines · Arnd Bergmann
Quoted 6 lines · Heiko Carstens
Quoted 5 lines · Danilo Krummrich
Quoted 3 lines · Arnd Bergmann

In theory you should be able to use rust code without PCI MMIO support, but I can't see any practical downsides to making rust 'depends on HAS_MMIO' to avoid having to add those #ifdef.

Quoted 8 lines
Quoted 8 lines

I think the implications should be minor without making Rust depend on CONFIG_HAS_IOMEM.

I just sent out a fix [1]; the only annoying part is [2], but we should change those doc-tests anyway. For the one in rust/kernel/io.rs we already did in driver-core-next.

[1] https://lore.kernel.org/driver-core/[email protected]/

This looks like you still provide the rust version of ioremap(), turning what is supposed to be a link failure into a runtime error.

Quoted 3 lines

I'm wondering if it would make sense to make HAS_IOMEM always available on s390, even though it doesn't make too much sense without PCI. But at least it would make s390 again a bit less special.

I see that with CONFIG_PCI=y, s390 already falls back to generic_ioremap_prot() and just maps any phys_addr_t into the page table as PAGE_KERNEL, regardless of whether this is an MMIO address or not.

The simple change below would just extend that behavior to !PCI and make that consistent with CONFIG_PCI=y on machines without actual PCI hardware. Of course any code that might rely on this is now a bug that likely never gets caught at build time.

Quoted 5 lines

This still relies on implementing the __raw_* helpers as nop to have the same behavior as the PCI=y version, as the generic version would just end up dereferencing the invalid pointers.

Arnd

I think there is also an interesting interaction with memremap(). That calls ioremap() under some circumstances and it's a known issue that on s390 memremap() will work if the PCI memory-I/O (MIO) instructions are enabled (e.g. on modern LPARs) because then ioremap() actually maps, but won't work if they aren't enabled. With your proposal it would also work for !PCI. Sadly, without working on the non-MIO one still really can't use memremap().

I actually have a prototype lying around where ioremap() always remaps by mapping the address cookies much like we do for user-space access via the s390 specific MMIO syscalls and then doing a page table walk in the accessors for the case where we don't have PCI MIO support. That would also allow us to implement the s390 MMIO syscall via generic_access_phys() getting rid of quite nasty inline assembly and over 300 lines removed in total. On the other hand it would cause overhead for non-MIO systems and sadly this currently includes all KVM and z/VM guests which is why it remains on my prototype pile. I haven't actually been able to measure the overhead but clearly more work is done.

Thanks,
Niklas

maintainer
#13 lore raw
Quoted 7 lines · Niklas Schnelle

I think there is also an interesting interaction with memremap(). That calls ioremap() under some circumstances and it's a known issue that on s390 memremap() will work if the PCI memory-I/O (MIO) instructions are enabled (e.g. on modern LPARs) because then ioremap() actually maps, but won't work if they aren't enabled. With your proposal it would also work for !PCI.

Right, that is an interesting one, as there are somewhat conflicting requirements:

- On PCI MMIO areas, memremap() and memremap_wt() should return a normal kernel pointer that can be dereferenced, e.g. for option ROM contents for a framebuffer console. This can't work on non-MIO guests but might work on MIO depending on which instructions are allowed those mappings (I can never quite remember how this part works on z, does this have to use pcistgi or does a normal aligned load/store work as well?).

- On areas that are backed by physical RAM but not part of the kernel memory (something like z/VM DCSS) should be mapped using memremap(), which would fail with the current !MIO implementation. On x86 and arm, this is done for mapping firmware blobs, but I don't think anything tries to do this on s390.
Right now, not having memremap() defined on !PCI configs is the one thing that lets you know if some non-PCI code ever start using it, other than crashing a non-MIO guest.

Quoted 11 lines

I actually have a prototype lying around where ioremap() always remaps by mapping the address cookies much like we do for user-space access via the s390 specific MMIO syscalls and then doing a page table walk in the accessors for the case where we don't have PCI MIO support. That would also allow us to implement the s390 MMIO syscall via generic_access_phys() getting rid of quite nasty inline assembly and over 300 lines removed in total. On the other hand it would cause overhead for non-MIO systems and sadly this currently includes all KVM and z/VM guests which is why it remains on my prototype pile. I haven't actually been able to measure the overhead but clearly more work is done.

This would not help with either memremap() or the !CONFIG_HAS_MMIO issue though, right?

Arnd

#14 lore raw
Quoted 12 lines · Arnd Bergmann
Quoted 7 lines · Niklas Schnelle

I think there is also an interesting interaction with memremap(). That calls ioremap() under some circumstances and it's a known issue that on s390 memremap() will work if the PCI memory-I/O (MIO) instructions are enabled (e.g. on modern LPARs) because then ioremap() actually maps, but won't work if they aren't enabled. With your proposal it would also work for !PCI.

Right, that is an interesting one, as there are somewhat conflicting requirements:

Quoted 7 lines

- On PCI MMIO areas, memremap() and memremap_wt() should return a normal kernel pointer that can be dereferenced, e.g. for option ROM contents for a framebuffer console. This can't work on non-MIO guests but might work on MIO depending on which instructions are allowed those mappings (I can never quite remember how this part works on z, does this have to use pcistgi or does a normal aligned load/store work as well?).

You still have to use pcistgi/pcilgi/pcistbi normal aligned loads/stores will fail as the physical address is beyond the memory limit. On the other hand the PCI instructions can't access normal memory. So you when using memremap() you'd have to know which kind of memory you're remapping and use the right accessors.

Quoted 23 lines

- On areas that are backed by physical RAM but not part of the kernel memory (something like z/VM DCSS) should be mapped using memremap(), which would fail with the current !MIO implementation. On x86 and arm, this is done for mapping firmware blobs, but I don't think anything tries to do this on s390.
Right now, not having memremap() defined on !PCI configs is the one thing that lets you know if some non-PCI code ever start using it, other than crashing a non-MIO guest.

Quoted 11 lines

I actually have a prototype lying around where ioremap() always remaps by mapping the address cookies much like we do for user-space access via the s390 specific MMIO syscalls and then doing a page table walk in the accessors for the case where we don't have PCI MIO support. That would also allow us to implement the s390 MMIO syscall via generic_access_phys() getting rid of quite nasty inline assembly and over 300 lines removed in total. On the other hand it would cause overhead for non-MIO systems and sadly this currently includes all KVM and z/VM guests which is why it remains on my prototype pile. I haven't actually been able to measure the overhead but clearly more work is done.

Quoted 4 lines

This would not help with either memremap() or the !CONFIG_HAS_MMIO issue though, right?

Arnd

I think it would help with memremap() because it would mean that when used on normal memory and accessed with normal loads/stores memremap() works and when used on PCI BAR / MIO addresses and accessed by I/O accessors it would also work. So it should behave the same no matter if memory-I/O is available.

Thanks,
Niklas

maintainer
#15 lore raw
Quoted 10 lines · Niklas Schnelle
Quoted 8 lines · Arnd Bergmann

On Tue, Aug 4, 2026, at 14:21, Niklas Schnelle wrote:
- On PCI MMIO areas, memremap() and memremap_wt() should return a normal kernel pointer that can be dereferenced, e.g. for option ROM contents for a framebuffer console. This can't work on non-MIO guests but might work on MIO depending on which instructions are allowed those mappings (I can never quite remember how this part works on z, does this have to use pcistgi or does a normal aligned load/store work as well?).

Quoted 5 lines

You still have to use pcistgi/pcilgi/pcistbi normal aligned loads/stores will fail as the physical address is beyond the memory limit. On the other hand the PCI instructions can't access normal memory. So you when using memremap() you'd have to know which kind of memory you're remapping and use the right accessors.

Ok, so memremap() by definition cannot work on s390 for regular PCI devices, since you are not allowed to use readl() etc on a kernel pointer returned by memremap(), only on an __iomem token returned by ioremap().

One important exception seems to be virtio_fs, which uses devm_memremap_pages() to map a virtio_shm_region, which in turn can come from a virtio-pci device but is backed by actual cached memory in the host instead of an emulated PCI memory BAR.

Quoted 8 lines
Quoted 2 lines

This would not help with either memremap() or the !CONFIG_HAS_MMIO issue though, right?

I think it would help with memremap() because it would mean that when used on normal memory and accessed with normal loads/stores memremap() works and when used on PCI BAR / MIO addresses and accessed by I/O accessors it would also work. So it should behave the same no matter if memory-I/O is available.

Have you come across any users of memremap() that pass the kernel address into readl()/writel() rather than accessing them as pointers? This would of course work on arm and x86, but is still a violation of the linux/io.h interface definition and causes a compile-time warning with sparse. If these exist, I think we just need to change them to ioremap().

Arnd

#16 lore raw
Quoted 22 lines · Arnd Bergmann
Quoted 10 lines · Niklas Schnelle
Quoted 8 lines · Arnd Bergmann

On Tue, Aug 4, 2026, at 14:21, Niklas Schnelle wrote:
- On PCI MMIO areas, memremap() and memremap_wt() should return a normal kernel pointer that can be dereferenced, e.g. for option ROM contents for a framebuffer console. This can't work on non-MIO guests but might work on MIO depending on which instructions are allowed those mappings (I can never quite remember how this part works on z, does this have to use pcistgi or does a normal aligned load/store work as well?).

Quoted 5 lines

You still have to use pcistgi/pcilgi/pcistbi normal aligned loads/stores will fail as the physical address is beyond the memory limit. On the other hand the PCI instructions can't access normal memory. So you when using memremap() you'd have to know which kind of memory you're remapping and use the right accessors.

Ok, so memremap() by definition cannot work on s390 for regular PCI devices, since you are not allowed to use readl() etc on a kernel pointer returned by memremap(), only on an __iomem token returned by ioremap().

Quoted 5 lines

One important exception seems to be virtio_fs, which uses devm_memremap_pages() to map a virtio_shm_region, which in turn can come from a virtio-pci device but is backed by actual cached memory in the host instead of an emulated PCI memory BAR.

Interesting, do I see that right that devm_memremap_pages() and memremap_pages() unlike plain memremap() does not utilize ioremap() and thus isn't affected by our nopped ioreamp() when there is no-MIO?

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

This would not help with either memremap() or the !CONFIG_HAS_MMIO issue though, right?

I think it would help with memremap() because it would mean that when used on normal memory and accessed with normal loads/stores memremap() works and when used on PCI BAR / MIO addresses and accessed by I/O accessors it would also work. So it should behave the same no matter if memory-I/O is available.

Quoted 8 lines

Have you come across any users of memremap() that pass the kernel address into readl()/writel() rather than accessing them as pointers? This would of course work on arm and x86, but is still a violation of the linux/io.h interface definition and causes a compile-time warning with sparse. If these exist, I think we just need to change them to ioremap().

Arnd

No, I'm not aware of any users of memremap() that would then use readl()/writel() and I agree they really should be using ioremap() and the use case doesn't really make sense I was just thinking about what would technically work with the prototype I have.

Thanks,
Niklas

maintainer
#17 lore raw
Quoted 13 lines · Niklas Schnelle
Quoted 11 lines · Arnd Bergmann

On Wed, Aug 5, 2026, at 17:36, Niklas Schnelle wrote:
Ok, so memremap() by definition cannot work on s390 for regular PCI devices, since you are not allowed to use readl() etc on a kernel pointer returned by memremap(), only on an __iomem token returned by ioremap().

One important exception seems to be virtio_fs, which uses devm_memremap_pages() to map a virtio_shm_region, which in turn can come from a virtio-pci device but is backed by actual cached memory in the host instead of an emulated PCI memory BAR.

Quoted 3 lines

Interesting, do I see that right that devm_memremap_pages() and memremap_pages() unlike plain memremap() does not utilize ioremap() and thus isn't affected by our nopped ioreamp() when there is no-MIO?

Indeed, I had not noticed that before, but you are right that this goes directly to the page tables while the other variants of memremap() go through architecture specific ioremap_prot(), which may or may not create page table entries.

I think the reason is that there are a couple of architectures (including non-MIO s390) that take a shortcut for normal ioremap() that avoids actually mapping anything, e.g. mips and related architectures have a fixed uncached mapping of all MMIO space visible to kernel, while some alpha, m68k powerpc, and s390 do something other than a pointer dereference for MMIO.

Arnd

#18 lore raw

Hi Arnd,

Quoted 7 lines · Arnd Bergmann

I think the reason is that there are a couple of architectures (including non-MIO s390) that take a shortcut for normal ioremap() that avoids actually mapping anything, e.g. mips and related architectures have a fixed uncached mapping of all MMIO space visible to kernel, while some alpha, m68k powerpc, and s390 do something other than a pointer dereference for MMIO.

FTR, several m68k platforms use the transparent translation registers to set up a mapping to the main system's MMIO space (like MIPS, but programmable), and thus just need to add an offset.

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#19 lore raw
In reply to Arnd Bergmann
Quoted 7 lines · Arnd Bergmann

I think the reason is that there are a couple of architectures (including non-MIO s390) that take a shortcut for normal ioremap() that avoids actually mapping anything, e.g. mips and related architectures have a fixed uncached mapping of all MMIO space visible to kernel, while some alpha, m68k powerpc, and s390 do something other than a pointer dereference for MMIO.

FWIW 32-bit MIPS configurations only have low 512MiB of MMIO accessible
via a fixed mapping and arrange for a regular mapping via the page table for anything beyond.

Maciej