x86: define arch_vm_get_page_prot to set _PAGE_IOMAP on VM_IO vmas
Subsystems (5)
PATCH 1/1 x86: define arch_vm_get_page_prot to set _PAGE_IOMAP on VM_IO vmas
Set _PAGE_IOMAP in ptes mapping a VM_IO vma. This says that the mapping is of a real piece of physical hardware, and not just system memory.
Xen, in particular, uses to this to inhibit the normal pfn->mfn conversion that would normally happen - in other words, treat the address directly as a machine physical address without converting it from pseudo-physical.
[ Impact: make VM_IO mappings map the right thing under Xen ]
arch/x86/include/asm/pgtable.h +3 -0
@@ -394,6 +394,9 @@ static inline unsigned long pages_to_mb(unsigned long npg)#define io_remap_pfn_range (vma, vaddr, pfn, size, prot) \ remap_pfn_range(vma, vaddr, pfn, size, prot) +#define arch_vm_get_page_prot arch_vm_get_page_prot +extern pgprot_t arch_vm_get_page_prot (unsigned vm_flags); +#if PAGETABLE_LEVELS >2 static inline int pud_none (pud_t pud) {
arch/x86/mm/pgtable.c +10 -0
@@ -6,6 +6,16 @@#define PGALLOC_GFP GFP_KERNEL | __GFP_NOTRACK | __GFP_REPEAT | __GFP_ZERO +pgprot_t arch_vm_get_page_prot (unsigned vm_flags) +{ +pgprot_t ret =__pgprot (0 ); + +if (vm_flags &VM_IO ) + ret =__pgprot (_PAGE_IOMAP); + +return ret; +} +pte_t *pte_alloc_one_kernel (struct mm_struct *mm,unsigned long address) {return (pte_t *)__get_free_page (PGALLOC_GFP );
From 81550b51436c282311c531d3ba7a79defd21c729 Mon Sep 17 00:00:00 2001
From: Jeremy Fitzhardinge <[email protected]>
Date: Fri, 20 Feb 2009 12:58:42 -0800
Subject: [PATCH] x86: define arch_vm_get_page_prot to set _PAGE_IOMAP on VM_IO vmas
Set _PAGE_IOMAP in ptes mapping a VM_IO vma. This says that the mapping is of a real piece of physical hardware, and not just system memory.
Xen, in particular, uses to this to inhibit the normal pfn->mfn conversion that would normally happen - in other words, treat the address directly as a machine physical address without converting it from pseudo-physical.
[ Impact: make VM_IO mappings map the right thing under Xen ]
arch/x86/include/asm/pgtable.h +3 -0
@@ -394,6 +394,9 @@ static inline unsigned long pages_to_mb(unsigned long npg)#define io_remap_pfn_range (vma, vaddr, pfn, size, prot) \ remap_pfn_range(vma, vaddr, pfn, size, prot) +#define arch_vm_get_page_prot arch_vm_get_page_prot +extern pgprot_t arch_vm_get_page_prot (unsigned vm_flags); +#if PAGETABLE_LEVELS >2 static inline int pud_none (pud_t pud) {
arch/x86/mm/pgtable.c +10 -0
@@ -6,6 +6,16 @@#define PGALLOC_GFP GFP_KERNEL | __GFP_NOTRACK | __GFP_REPEAT | __GFP_ZERO +pgprot_t arch_vm_get_page_prot (unsigned vm_flags) +{ +pgprot_t ret =__pgprot (0 ); + +if (vm_flags &VM_IO ) + ret =__pgprot (_PAGE_IOMAP); + +return ret; +} +pte_t *pte_alloc_one_kernel (struct mm_struct *mm,unsigned long address) {return (pte_t *)__get_free_page (PGALLOC_GFP );
H. Peter Anvin
reviewerQuoted 4 lines · Jeremy Fitzhardinge
Set
_PAGE_IOMAPin ptes mapping aVM_IOvma. This says that the mapping is of a real piece of physical hardware, and not just system memory.
Quoted 8 lines
Xen, in particular, uses to this to inhibit the normal
pfn->mfnconversion that would normally happen - in other words, treat the address directly as a machine physical address without converting it from pseudo-physical.[ Impact: make
VM_IOmappings map the right thing under Xen ]Signed-off-by Jeremy Fitzhardinge <[email protected]>
Am I the only one who thinks this seems extremely odd that the guest is trusted to make this distinction?
-hpa
Quoted 8 lines · H. Peter Anvin
Quoted 7 lines · Jeremy Fitzhardinge
Set
_PAGE_IOMAPin ptes mapping aVM_IOvma. This says that the mapping is of a real piece of physical hardware, and not just system memory.Xen, in particular, uses to this to inhibit the normal
pfn->mfnconversion that would normally happen - in other words, treat the address directly as a machine physical address without converting it from pseudo-physical.
Quoted 6 lines
Quoted 4 lines
[ Impact: make
VM_IOmappings map the right thing under Xen ]Signed-off-by Jeremy Fitzhardinge <[email protected]>Am I the only one who thinks this seems extremely odd that the guest is trusted to make this distinction?
Xen PV guests are always responsible for constructing ptes with machine addresses in them (ie, doing their own pseudo-physical to machine address conversion), and Xen verifies that the pages they want to map either belong to them or have been granted to them. The _PAGE_IOMAP flag is a kernel-internal one which allows us to distinguish between ptes intended to map memory vs machine hardware addresses; it is not part of the Xen ABI.
If you're passing a device through to a domain, the domain is given access to the device's address space so it can legally map those pages (and if an IOMMU is available, the device is constrained to only DMA that domain's memory).
J
H. Peter Anvin
reviewerQuoted 9 lines · Jeremy Fitzhardinge
Xen PV guests are always responsible for constructing ptes with machine addresses in them (ie, doing their own pseudo-physical to machine address conversion), and Xen verifies that the pages they want to map either belong to them or have been granted to them. The
_PAGE_IOMAPflag is a kernel-internal one which allows us to distinguish between ptes intended to map memory vs machine hardware addresses; it is not part of the Xen ABI.
Quoted 5 lines
If you're passing a device through to a domain, the domain is given access to the device's address space so it can legally map those pages (and if an IOMMU is available, the device is constrained to only DMA that domain's memory).
Okay, could you clarify this part a bit? Why does the kernel need to know the difference between "pseudo-physical" and "machine addresses" at all? If they overlap, there is a problem, and if they don't overlap, it will be a 1:1 mapping anyway...
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 16 lines · H. Peter Anvin
Quoted 9 lines · Jeremy Fitzhardinge
Xen PV guests are always responsible for constructing ptes with machine addresses in them (ie, doing their own pseudo-physical to machine address conversion), and Xen verifies that the pages they want to map either belong to them or have been granted to them. The
_PAGE_IOMAPflag is a kernel-internal one which allows us to distinguish between ptes intended to map memory vs machine hardware addresses; it is not part of the Xen ABI.Quoted 5 lines
If you're passing a device through to a domain, the domain is given access to the device's address space so it can legally map those pages (and if an IOMMU is available, the device is constrained to only DMA that domain's memory).
Quoted 4 lines
Okay, could you clarify this part a bit? Why does the kernel need to know the difference between "pseudo-physical" and "machine addresses" at all? If they overlap, there is a problem, and if they don't overlap, it will be a 1:1 mapping anyway...
The flag (_PAGE_IOMAP) is used when we set the PTE so that the MFN value is used instead of the PFN. We need that b/c when a driver does page_to_pfn() it ends up using the PFN as bus address to write out registers data.
Without this patch, the page->virt->PFN value is used and the PFN != to real MFN so we end up writing in a memory address that the PCI device has no idea about. By setting the PTE with the MFN, the virt->PFN gets the real MFN value.
The drivers I am talking about are mostly, if not all, located in drivers/gpu and it looks that we are missing two more patches to utilize the patch that Jeremy posted.
Please note that I am _not_ suggesting that the two patches below should go out - I still need to post them on drm mailing list.
commit 1b5a6e09831f44445cdb96f29c287e18d0317136
Author: Jeremy Fitzhardinge <[email protected]> Date: Fri Oct 2 09:49:05 2009 -0700
drm: recompute vma->vm_page_prot after changing vm_flags
vm_get_page_prot() computes vm_page_prot depending on vm_flags, so we need to re-call it if we change flags.
drivers/gpu/drm/ttm/ttm_bo_vm.c +2 -0
@@ -272,6 +272,7 @@ int ttm_bo_mmap(struct file *filp, struct vm_area_struct *vma, vma->vm_private_data = bo; vma->vm_flags |=VM_RESERVED |VM_IO |VM_MIXEDMAP |VM_DONTEXPAND ; + vma->vm_page_prot =vm_get_page_prot (vma->vm_flags);return 0 ; out_unref:ttm_bo_unref (&bo);
@@ -287,6 +288,7 @@ int ttm_fbdev_mmap(struct vm_area_struct *vma, struct ttm_buffer_object *bo) vma->vm_ops = &ttm_bo_vm_ops; vma->vm_private_data =ttm_bo_reference (bo); vma->vm_flags |=VM_RESERVED |VM_IO |VM_MIXEDMAP |VM_DONTEXPAND ; + vma->vm_page_prot =vm_get_page_prot (vma->vm_flags);return 0 ; }EXPORT_SYMBOL (ttm_fbdev_mmap);
commit 9551827190db2d34f106255f0b5bfdc452e85cd8
Author: Konrad Rzeszutek Wilk <[email protected]> Date: Mon Jul 26 14:15:55 2010 -0400
ttm: Set VM_IO only on pages with TTM_MEMTYPE_FLAG_NEEDS_IOREMAP set.
This patch is based on "[Patch RFC] ttm: nouveau accelerated on Xen pv-ops kernel"
http://lists.freedesktop.org/archives/nouveau/2010-March/005326.html
Under Xen, the PFN of page is virtualized. The physical addresses used for DMA programming needs to be the Machine Frame Number (MFN). Xen transparently does the correct translation using the _PAGE_IOMEM PTE bit. If the bit is set, Xen assumes that the backing memory is in the IOMEM space, and PFN equals MFN. If not set, page_to_pfn() returns a phantom MFN.
The patch enables the ttm_bo_vm_fault() handler to behave correctly under Xen, and has no side-effects on normal (not under Xen) operations.
The use of TTM_MEMTYPE_FLAG_NEEDS_IOREMAP in the check assumes that only pages which have this flag are backed by device memory or IO.
drivers/gpu/drm/ttm/ttm_bo_vm.c +5 -2
@@ -239,6 +239,7 @@ int ttm_bo_mmap(struct file *filp, struct vm_area_struct *vma, {struct ttm_bo_driver *driver;struct ttm_buffer_object *bo; +struct ttm_mem_type_manager *man;int ret;read_lock (&bdev->vm_lock);
@@ -271,7 +272,10 @@ int ttm_bo_mmap(struct file *filp, struct vm_area_struct *vma,*/ vma->vm_private_data = bo; - vma->vm_flags |=VM_RESERVED |VM_IO |VM_MIXEDMAP |VM_DONTEXPAND ; + vma->vm_flags |=VM_RESERVED |VM_MIXEDMAP |VM_DONTEXPAND ; + man = &bdev->man[bo->mem.mem_type]; +if (man->flags &TTM_MEMTYPE_FLAG_NEEDS_IOREMAP ) + vma->vm_flags |=VM_IO ; vma->vm_page_prot =vm_get_page_prot (vma->vm_flags);return 0 ; out_unref:
@@ -288,7 +292,6 @@ int ttm_fbdev_mmap(struct vm_area_struct *vma, struct ttm_buffer_object *bo) vma->vm_ops = &ttm_bo_vm_ops; vma->vm_private_data =ttm_bo_reference (bo); vma->vm_flags |=VM_RESERVED |VM_IO |VM_MIXEDMAP |VM_DONTEXPAND ; - vma->vm_page_prot =vm_get_page_prot (vma->vm_flags);return 0 ; }EXPORT_SYMBOL (ttm_fbdev_mmap);
Quoted 12 lines
-hpa
Signature · 8 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
H. Peter Anvin
reviewerQuoted 14 lines · Konrad Rzeszutek Wilk
Quoted 5 lines
Okay, could you clarify this part a bit? Why does the kernel need to know the difference between "pseudo-physical" and "machine addresses" at all? If they overlap, there is a problem, and if they don't overlap, it will be a 1:1 mapping anyway...
The flag (
_PAGE_IOMAP) is used when we set the PTE so that the MFN value is used instead of the PFN. We need that b/c when a driver doespage_to_pfn()it ends up using the PFN as bus address to write out registers data.Without this patch, the
page->virt->PFNvalue is used and the PFN != to real MFN so we end up writing in a memory address that the PCI device has no idea about. By setting the PTE with the MFN, thevirt->PFNgets the real MFN value.
Quoted 7 lines
The drivers I am talking about are mostly, if not all, located in
drivers/gpuand it looks that we are missing two more patches to utilize the patch that Jeremy posted.Please note that I am _not_ suggesting that the two patches below should go out - I still need to post them on drm mailing list.
I'm still seriously confused. If I understand this correctly, we're talking about DMA addresses here (as opposed to PIO addresses, i.e. BARs), right?
It's the bimodality that really bothers me. I understand of course that Xen imposes yet another address remapping layer, but I'm having a hard time understanding any conditions under with we would need that layer to go away, as long as DMA addresses are translated via the DMA APIs -- and if they aren't, then iommus will break, too.
As such, I don't grok this page flag and what it does, and why it's needed. I'm not saying it's *wrong*, I'm saying the design is opaque to me and I'm not sure it is the right solution.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 19 lines · H. Peter Anvin
Quoted 18 lines · Konrad Rzeszutek Wilk
Quoted 5 lines
Okay, could you clarify this part a bit? Why does the kernel need to know the difference between "pseudo-physical" and "machine addresses" at all? If they overlap, there is a problem, and if they don't overlap, it will be a 1:1 mapping anyway...
The flag (
_PAGE_IOMAP) is used when we set the PTE so that the MFN value is used instead of the PFN. We need that b/c when a driver doespage_to_pfn()it ends up using the PFN as bus address to write out registers data.Without this patch, the
page->virt->PFNvalue is used and the PFN != to real MFN so we end up writing in a memory address that the PCI device has no idea about. By setting the PTE with the MFN, thevirt->PFNgets the real MFN value.The drivers I am talking about are mostly, if not all, located in
drivers/gpuand it looks that we are missing two more patches to utilize the patch that Jeremy posted.
Quoted 7 lines
Quoted 3 lines
Please note that I am _not_ suggesting that the two patches below should go out - I still need to post them on drm mailing list.
I'm still seriously confused. If I understand this correctly, we're talking about DMA addresses here (as opposed to PIO addresses, i.e. BARs), right?
Correct. The BARs are ok since they go through the ioremap.
Quoted 6 lines
It's the bimodality that really bothers me. I understand of course that Xen imposes yet another address remapping layer, but I'm having a hard time understanding any conditions under with we would need that layer to go away, as long as DMA addresses are translated via the DMA APIs -- and if they aren't, then iommus will break, too.
That is it. They aren't using the DMA or PCI API completly(*). Try doing 'iommu=soft swiotlb=force' with your radeon card under baremetal
(I used an ATI ES1000). I think it will grind to halt during the writeback test.
(*): This was with 2.6.34, I haven't touched 2.6.36 and there was an drm/iomem rewrite so it might be that this now working. The incomplete part of the graphics drivers was that it would not do pci_dma_sync_*, so when the MFN was programmed in the GTT/GART (check out radeon_gart_bind, the call to pci_map_page gets the bus address, also known as MFN). So the GPU would now have a virt->MFN mapping. However, on the CPU side when the driver writes a texture to virtual address, the mapping is virt->PFN.
So when we kick the GPU to do its magic, the VM on the graphics card would translate the virtual address to the MFN, which did not have the data that was written by the kernel to the PFN. In other words *PFN != *MFN, while we need *PFN == *MFN. There are two ways of making this work:
1). PFN == MFN (this is what Jeremy's patch ends up doing) and under
baremetal it won't affect as baremetal doesn't care what the VM_IO flag stands for.
2). Add a whole bunch of pci_dma_sync in the appropiate sections in the
graphic drivers.
I am not qualified to do 2) - that code scares me. Also 1) is the easier :-)
I am actually not sure how it works with AMD-Vi or Intel VT-d. I do remember something about letting certain devices bypass the VT-d, and I think I saw the nouveau making the DMAR throw a fit.
Quoted 2 lines
As such, I don't grok this page flag and what it does, and why it's needed. I'm not saying it's *wrong*, I'm saying the design is opaque to
I hope my explanation cleared the confusion.
Quoted 1 line
me and I'm not sure it is the right solution.
Quoted 24 lines · Konrad Rzeszutek Wilk
Quoted 23 lines · H. Peter Anvin
Quoted 21 lines · Konrad Rzeszutek Wilk
Quoted 5 lines
Okay, could you clarify this part a bit? Why does the kernel need to know the difference between "pseudo-physical" and "machine addresses" at all? If they overlap, there is a problem, and if they don't overlap, it will be a 1:1 mapping anyway...
The flag (
_PAGE_IOMAP) is used when we set the PTE so that the MFN value is used instead of the PFN. We need that b/c when a driver doespage_to_pfn()it ends up using the PFN as bus address to write out registers data.Without this patch, the
page->virt->PFNvalue is used and the PFN != to real MFN so we end up writing in a memory address that the PCI device has no idea about. By setting the PTE with the MFN, thevirt->PFNgets the real MFN value.The drivers I am talking about are mostly, if not all, located in
drivers/gpuand it looks that we are missing two more patches to utilize the patch that Jeremy posted.Please note that I am _not_ suggesting that the two patches below should go out - I still need to post them on drm mailing list.
Quoted 5 lines
Quoted 3 lines
I'm still seriously confused. If I understand this correctly, we're talking about DMA addresses here (as opposed to PIO addresses, i.e. BARs), right?
Correct. The BARs are ok since they go through the ioremap.
Whoops. I answered that incorrectly. No, we are not talking about BARs. The BARs are ok (look at x86/PCI: make sure _PAGE_IOMAP it set on pci mappings patch, http://amailbox.org/mailarchive/linux-kernel/2010/10/12/4630930/thread)
We are talking about the GPU's VM engine (or the GART on the Northbridge).
H. Peter Anvin
reviewerQuoted 4 lines · Konrad Rzeszutek Wilk
I hope my explanation cleared the confusion.
Quoted 1 line
me and I'm not sure it is the right solution.
OK, now I'm *positive* this isn't the right solution...
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 5 lines
Quoted 3 lines
I hope my explanation cleared the confusion.
Quoted 1 line
me and I'm not sure it is the right solution.
OK, now I'm *positive* this isn't the right solution...
Uhhh.. could you elaborate on a right solution?
Quoted 30 lines · H. Peter Anvin
Quoted 12 lines · Konrad Rzeszutek Wilk
Quoted 4 lines
Okay, could you clarify this part a bit? Why does the kernel need to know the difference between "pseudo-physical" and "machine addresses" at all? If they overlap, there is a problem, and if they don't overlap, it will be a 1:1 mapping anyway...
The flag (
_PAGE_IOMAP) is used when we set the PTE so that the MFN value is used instead of the PFN. We need that b/c when a driver doespage_to_pfn()it ends up using the PFN as bus address to write out registers data.Without this patch, the
page->virt->PFNvalue is used and the PFN != to real MFN so we end up writing in a memory address that the PCI device has no idea about. By setting the PTE with the MFN, thevirt->PFNgets the real MFN value.Quoted 7 lines
The drivers I am talking about are mostly, if not all, located in
drivers/gpuand it looks that we are missing two more patches to utilize the patch that Jeremy posted.Please note that I am _not_ suggesting that the two patches below should go out - I still need to post them on drm mailing list.
I'm still seriously confused. If I understand this correctly, we're talking about DMA addresses here (as opposed to PIO addresses, i.e. BARs), right?
It's the bimodality that really bothers me. I understand of course that Xen imposes yet another address remapping layer, but I'm having a hard time understanding any conditions under with we would need that layer to go away, as long as DMA addresses are translated via the DMA APIs -- and if they aren't, then iommus will break, too.
Quoted 3 lines
As such, I don't grok this page flag and what it does, and why it's needed. I'm not saying it's *wrong*, I'm saying the design is opaque to me and I'm not sure it is the right solution.
Well, if you want to map a normal memory page, you'd use, say, pfn_pte(pfn, PAGE_KERNEL) to generate the pte. The pfn is a domain-local pseudo-physical address. When it ends up in xen_make_pte(), it will translate the the pfn into a machine-global mfn to generate a pte_t which can be inserted into a pagetable. (And when that pagetable starts being used as such, Xen will validate that the mfn is actually one the domain is allowed to address.)
However, if you're doing an ioremap(), then the mapped address is a hardware one. In that case, we construct the pte with pfn_pte(device_pfn, PAGE_KERNEL_IO), which sets the _PAGE_IOMAP flag in the pte flags. When it gets to xen_make_pte(), it sees _PAGE_IOMAP and constructs a pte_t containing the literal untranslated device_pfn (really an mfn). (And again, Xen will check that the domain has access to that mfn before allowing the mapping to be used.)
We use the DMA API to make sure that pfn<->mfn conversions happen correctly when setting up DMA (all the Xen swiotlb stuff Konrad has been working on). _PAGE_IOMAP is used in the implementation of that, as well as in ioremap(), remap_pfn_range(), drm, etc.
All this machinery has been in the kernel for quite a while. This particular patch fixes gap, making sure that a vma with VM_IO set will be mapped with _PAGE_IOMAP set, which makes a large class of drm, fbdev, capture, etc device drivers work properly under Xen unmodified. (Though DRM is full of other pitfalls and Konrad has been up to his neck in piranhas lately, which is why is answer is a little fixated on the DRM aspect.)
J
H. Peter Anvin
reviewerQuoted 9 lines · Jeremy Fitzhardinge
Well, if you want to map a normal memory page, you'd use, say,
pfn_pte(pfn, PAGE_KERNEL)to generate the pte. The pfn is a domain-local pseudo-physical address. When it ends up inxen_make_pte(), it will translate the the pfn into a machine-global mfn to generate apte_twhich can be inserted into a pagetable. (And when that pagetable starts being used as such, Xen will validate that the mfn is actually one the domain is allowed to address.)
Quoted 8 lines
However, if you're doing an
ioremap(), then the mapped address is a hardware one. In that case, we construct the pte withpfn_pte(device_pfn, PAGE_KERNEL_IO), which sets the_PAGE_IOMAPflag in the pte flags. When it gets toxen_make_pte(), it sees_PAGE_IOMAPand constructs apte_tcontaining the literal untranslateddevice_pfn(really an mfn). (And again, Xen will check that the domain has access to that mfn before allowing the mapping to be used.)
When you're doing an ioremap(), then the mapped address is *both* a PFN and an MFN, right? So why do your need a flag? That is the part I don't get...
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 17 lines · H. Peter Anvin
Quoted 8 lines · Jeremy Fitzhardinge
Well, if you want to map a normal memory page, you'd use, say,
pfn_pte(pfn, PAGE_KERNEL)to generate the pte. The pfn is a domain-local pseudo-physical address. When it ends up inxen_make_pte(), it will translate the the pfn into a machine-global mfn to generate apte_twhich can be inserted into a pagetable. (And when that pagetable starts being used as such, Xen will validate that the mfn is actually one the domain is allowed to address.)Quoted 8 lines
However, if you're doing an
ioremap(), then the mapped address is a hardware one. In that case, we construct the pte withpfn_pte(device_pfn, PAGE_KERNEL_IO), which sets the_PAGE_IOMAPflag in the pte flags. When it gets toxen_make_pte(), it sees_PAGE_IOMAPand constructs apte_tcontaining the literal untranslateddevice_pfn(really an mfn). (And again, Xen will check that the domain has access to that mfn before allowing the mapping to be used.)
Quoted 3 lines
When you're doing an
ioremap(), then the mapped address is *both* a PFN and an MFN, right? So why do your need a flag? That is the part I don't get...
Xen always needs an mfn, so for memory the pfn needs to be converted to an mfn, but for devices the frame number is already an mfn. You could look at it as having two distinct frame number address spaces, and the _PAGE_IOMAP flag tells the lower levels which address space the frame number is in. Both memory and device mappings use the same interface - __pte() - to convert a (pfn,prot) tuple into a pte_t, so we use a bit in prot to distinguish them.
In the kernel, the places where device mappings vs memory mappings are quite well delineated. There used to be a few places which could be either depending on context, but they've since been cleaned up by other efforts (the PAT/pageattr work did a lot of them, I think).
J