Skip to contentSkip to conversation

mmap crashes system after mapping PCI device into user space and reading and writing.

Linux-ia64 Mike Fox 4 participants 9 messages Apr 8, 2003 – Apr 10, 2003

lore
Share & tools
lore

Mike Fox

author
#0 lore raw

Hi,

I am running kernel version 2.4.19 and I am getting a system crash when I mmap my PCI device into user space and access it.

It seems to be the same issue that was posted earlier in the following post, which had no response:

Has anyone tried using mmap to remap PCI device memory into user space? I have seen the following problem: If I do the mmap, it returns successfully, and I can indeed access the device memory. Unfortunately, the system will hang within about a minute after doing so. I do not get an oops, or crash of any kind, simply no further response form the machine. If I just do the mmap and then munmap the system will not hang. But if I do one access to the mmapped address, the system will hang. I am using the remap_page_range call in my mmap routine, and this same code works on x86 and Alpha platforms. I found it odd that it takes a while for the system to hang after the access, is their something special I have to do for this to work on IA64? I am running 2.4.1 with the 1/31 patch, although this problem occurs on all previous kernels I have tried. Thanks in advance for any insight anyone might have.

Thanks for any help you can give. I'm new to Linux and IA64 and I am in a rush to port a PCI driver and API.

Thanks,

Mike Fox.

#1 lore raw

Re: [Linux-ia64] mmap crashes system after mapping PCI device into user space and reading and writin

On Tue, Apr 08, 2003 at 05:59:56PM -0400, Mike Fox wrote:
...

Quoted 4 lines

Has anyone tried using mmap to remap PCI device memory into user space? I have seen the following problem: If I do the mmap, it returns successfully, and I can indeed access the device memory. Unfortunately, the system will hang within about a minute after doing so.

...

David Mosberger passed on a very short program who's essence is:
mem = mmap (NULL, page_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, offset); if (mem = MAP_FAILED)

  {
    perror ("memmap");
    exit (-1);

}

This works on the HP ZX1 platforms with 2.4.20+patches kernel.

Be warned, it's *very* easy to crash a machine when using /dev/mem and scribbling in misc parts of IO space. The symptom will typically be an MCA (see "errdump MCA" output from EFI to get the xip).

grant

author
#2 lore raw

RE: [Linux-ia64] mmap crashes system after mapping PCI device into user space and reading and writin

In reply to Mike Fox

Ok I have some more info. The crash only happens after I do a write to my PCI controller. I map the PCI controller into user space and I can read and write a scratch register ok. I can also read other initialized registers to confirm I am reading my PCI controller. I can do many reads, but after some number of writes the system goes down. I went ahead and updated the kernel to 2.4.20 with the latest patches. Any thoughts or a push in the right direction are very much appreciated.

My mmap call in user space looks something like this:

pRegRegionMappedAddr = (DWORD*)mmap( 0,
WSII_REG_SPACE_SIZE_BYTES, PROT_WRITE | PROT_READ, MAP_SHARED, DriverHandle,
0 );

In the driver I am basically doing this in my mmap entry point:

  vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
  vma->vm_flags |= VM_IO;

/*
* this maps the physical memory to the user
*/
if (remap_page_range(vma->vm_start, offset, vma->vm_end-vma->vm_start,

                            vma->vm_page_prot))
  {
    return -EAGAIN;

}

Thanks,
Mike.

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Grant Grundler
Sent: Tuesday, April 08, 2003 8:43 PM
To: Mike Fox
Cc: [email protected]
Subject: Re: [Linux-ia64] mmap crashes system after mapping PCI device into user space and reading and writing.

On Tue, Apr 08, 2003 at 05:59:56PM -0400, Mike Fox wrote:
...

Quoted 2 lines

Has anyone tried using mmap to remap PCI device memory into user space? I have seen the following problem: If I do the mmap, it returns

successfully,

Quoted 2 lines

and I can indeed access the device memory. Unfortunately, the system will hang within about a minute after doing so.

...

David Mosberger passed on a very short program who's essence is:
mem = mmap (NULL, page_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd,
offset);
if (mem == MAP_FAILED)

  {
    perror ("memmap");
    exit (-1);

}

This works on the HP ZX1 platforms with 2.4.20+patches kernel.

Be warned, it's *very* easy to crash a machine when using /dev/mem and scribbling in misc parts of IO space. The symptom will typically be an MCA (see "errdump MCA" output from EFI to get the xip).

grant

_______________________________________________
Linux-IA64 mailing list
[email protected]
http://lists.linuxia64.org/lists/listinfo/linux-ia64

#3 lore raw

Re: [Linux-ia64] mmap crashes system after mapping PCI device into user space and reading and writin

In reply to Mike Fox
Quoted 8 lines · Mike Fox

/*
* this maps the physical memory to the user
*/
if (remap_page_range(vma->vm_start, offset, vma->vm_end-vma->vm_start,

                            vma->vm_page_prot))
  {
    return -EAGAIN;

}

The example code I posted (a) used physical addresses and (b) assume the "user" was root (to access /dev/mem/).

XFree86 pokes around in PCI MMIO and IO space and does work. Have you tried to dig through the source of XF86 4.2?

Have you checked the return address (to user space) is uncached? If I understood details of VM it might be obvious from the code you posted.

hth,
grant

Quoted 51 lines

Thanks,
Mike.

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Grant Grundler
Sent: Tuesday, April 08, 2003 8:43 PM
To: Mike Fox
Cc: [email protected]
Subject: Re: [Linux-ia64] mmap crashes system after mapping PCI device into user space and reading and writing.

On Tue, Apr 08, 2003 at 05:59:56PM -0400, Mike Fox wrote:
...

Quoted 2 lines

Has anyone tried using mmap to remap PCI device memory into user space? I have seen the following problem: If I do the mmap, it returns

successfully,

Quoted 2 lines

and I can indeed access the device memory. Unfortunately, the system will hang within about a minute after doing so.

...

David Mosberger passed on a very short program who's essence is:
mem = mmap (NULL, page_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd,
offset);
if (mem = MAP_FAILED)

  {
    perror ("memmap");
    exit (-1);

}

This works on the HP ZX1 platforms with 2.4.20+patches kernel.

Be warned, it's *very* easy to crash a machine when using /dev/mem and scribbling in misc parts of IO space. The symptom will typically be an MCA (see "errdump MCA" output from EFI to get the xip).

grant

_______________________________________________
Linux-IA64 mailing list
[email protected]
http://lists.linuxia64.org/lists/listinfo/linux-ia64

_______________________________________________
Linux-IA64 mailing list
[email protected]
http://lists.linuxia64.org/lists/listinfo/linux-ia64

Signature · 2 lines
Grant Grundler                           Linux Open Systems Lab/BCST
[email protected]                      +1.408.447.7253
#4 lore raw

Re: [Linux-ia64] mmap crashes system after mapping PCI device into user

In reply to Mike Fox

Hi,

Quoted 6 lines · Mike Fox

In the driver I am basically doing this in my mmap entry point:

  vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
  vma->vm_flags |= VM_IO;

i know this is ugly stuff, but ...
maybe you want to try this instead of the above:

#if defined(__ia64__)
	vmap->vm_flags |= (VM_SHM | VM_LOCKED | VM_IO);
	vmap->vm_flags &= ~(VM_SHARED | VM_MAYSHARE);
	vmap->vm_page_prot = __pgprot(pgprot_val(vmap->vm_page_prot)& ~_PAGE_AR_RW);

#endif

Quoted 4 lines

Thanks,
Mike.

christian

#5 lore raw

Re: [Linux-ia64] mmap crashes system after mapping PCI device into user space and reading and writin

In reply to Mike Fox
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Mike Fox

Mike> Hi, I am running kernel version 2.4.19 and I am getting a Mike> system crash when I mmap my PCI device into user space and Mike> access it.

One thing you need to be careful about is memory attribute aliasing. The CPU (or platform chips) can MCA if you try to access the same physical location with different memory attributes (e.g., cached vs. uncached). Also, keep in mind that the kernel uses a 64MB page size for the identity-mapped regions (regions 6 and 7). This means that you cannot map stuff requiring different attributes within the same 64MB page. I kind of doubt this is what's causing the problem for you, but since you didn't give any details about the hw you're trying to map, it's hard to be more specific.

As others have pointed out, it's of course also important to verify that you access I/O space uncached.

	--david
#6 lore raw

RE: [Linux-ia64] mmap crashes system after mapping PCI device into user space and reading and writin

In reply to Mike Fox
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Mike Fox

Mike> In the driver I am basically doing this in my mmap entry point:

  Mike>   vma-> vm_page_prot = pgprot_noncached(vma->vm_page_prot);
  Mike>   vma-> vm_flags |= VM_IO;

Mike> /*
Mike> * this maps the physical memory to the user
Mike> */
Mike> if (remap_page_range(vma->vm_start, offset, vma->vm_end-vma->vm_start, vma->vm_page_prot)) Mike> {
Mike> return -EAGAIN;
Mike> }

This is basically what drivers/char/mem.c is doing, so there is nothing obviously wrong here.

	--david
author
#7 lore raw

RE: [Linux-ia64] mmap crashes system after mapping PCI device into user space and reading and writin

In reply to Mike Fox

Hey all, I got it working! I was porting this code from a 32 bit version, and it was as follows:

/*
* Accessing memory above the top the kernel knows about or
* through a file pointer that was marked O_SYNC will be
* done non-cached.
*
* Set VM_IO, as this is likely a non-cached access to an
* I/O area, and we don't want to include that in a core
* file.
*/
if (offset >= __pa(high_memory) || (file->f_flags & O_SYNC))

  {
     vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
     vma->vm_flags |= VM_IO;

}

Turns out my pgprot_noncached was not getting called, so I took it out of the conditional. Thanks for all of your help.

I just started working at my current company 3 weeks ago, but it looks like we will be doing a lot of work in the Linux environment on 32 and 64 bit architectures.

I went ahead and bought the book, "ia-64 linux kernel", by David Mosberger and Stephane Eranian. My background is in EE, and I never got a chance to take an OS class, but I figure this book is my chance.

Thanks again,
Mike Fox.

-----Original Message-----
From: David Mosberger [mailto:[email protected]]
Sent: Wednesday, April 09, 2003 1:46 PM
To: [email protected]
Cc: [email protected]
Subject: RE: [Linux-ia64] mmap crashes system after mapping PCI device into user space and reading and writing.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

On Wed, 9 Apr 2003 01:36:33 -0400, "Mike Fox" <[email protected]>

said:

Mike> In the driver I am basically doing this in my mmap entry point:

  Mike>   vma-> vm_page_prot = pgprot_noncached(vma->vm_page_prot);
  Mike>   vma-> vm_flags |= VM_IO;

Mike> /*
Mike> * this maps the physical memory to the user Mike> */
Mike> if (remap_page_range(vma->vm_start, offset,
vma->vm_end-vma->vm_start, vma->vm_page_prot))
Mike> {
Mike> return -EAGAIN;
Mike> }

This is basically what drivers/char/mem.c is doing, so there is nothing obviously wrong here.

	--david
#8 lore raw

RE: [Linux-ia64] mmap crashes system after mapping PCI device into user space and reading and writin

In reply to Mike Fox
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Mike Fox

Mike> Turns out my pgprot_noncached was not getting called, so I Mike> took it out of the conditional. Thanks for all of your help.

Ah, that explains it.

Mike> I went ahead and bought the book, "ia-64 linux kernel", by Mike> David Mosberger and Stephane Eranian.

Great choice! ;-))

Seriously though: an errata, kernel change log, and a searchable index are available at:

http://www.lia64.org/book/

There were some stupid typos that snuck by, so the errata is especially important.

	--david