Skip to contentSkip to conversation

Special handling of sysfs device resource files?

DISCUSSION Ian Romanick 3 participants 6 messages Apr 11, 2006 – Apr 12, 2006

lore
Share & tools
lore
#0 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I'm in the process of modifying X to be civilized in it's handling of PCI devices on Linux. As part of that, I've modified it to map the /sys/bus/pci/device/*/resource[0-6] files instead of mucking about with /dev/mem.

This seems to mostly work, but I am having one problem. I map the region by opening the file with O_RDWR, then mmap with

(PROT_READ|PROT_WRITE) and MAP_SHARED.  In all cases, the open and mmap
succeed.  However, for I/O BARs, the resulting pointer from mmap is
invalid.  Any access to it results in a segfault and GDB says it's "out

of range".

The base address of the BAR is page aligned, so its not a problem with the alignment of mmap vs. the alignment of the BAR. What else could it be? I'm pretty stumped.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (GNU/Linux)

iD8DBQFEPB7KX1gOwKyEAw8RAhG/AJ4x+Vjl8V9SNeyMhYe2txeAeKALKACePCwL 6s0kj4YhDY3/thVh6mvO5X4=
=g9eu
-----END PGP SIGNATURE-----

#1 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 11 lines · Ian Romanick

I'm in the process of modifying X to be civilized in it's handling of PCI devices on Linux. As part of that, I've modified it to map the /sys/bus/pci/device/*/resource[0-6] files instead of mucking about with /dev/mem.

This seems to mostly work, but I am having one problem. I map the region by opening the file with O_RDWR, then mmap with

(PROT_READ|PROT_WRITE) and MAP_SHARED.  In all cases, the open and mmap
succeed.  However, for I/O BARs, the resulting pointer from mmap is
invalid.  Any access to it results in a segfault and GDB says it's "out

of range".

I was a little mistaken about this. The BAR that causes the problem is not I/O. It *is* memory.

01:00.0 VGA compatible controller: Matrox Graphics, Inc. G400/G450 (rev
03) (prog-if 00 [VGA])
Subsystem: Matrox Graphics, Inc. Millennium G400 16Mb SGRAM
Flags: bus master, medium devsel, latency 64, IRQ 11 Memory at cc000000 (32-bit, prefetchable) [size=32M] Memory at cfefc000 (32-bit, non-prefetchable) [size=16K] Memory at cf000000 (32-bit, non-prefetchable) [size=8M] Expansion ROM at cfee0000 [disabled] [size=64K]

When I open and mmap resource0 (the framebuffer) I get 0x2b9aa48ea000. When I open and mmap resource1 (the card's registers) I get 0x2b9aa68ea000. I can access the resource0 pointer all day long without problems. The firs access to the resource1 pointer results in a segfault.

Quoted 3 lines

The base address of the BAR is page aligned, so its not a problem with the alignment of mmap vs. the alignment of the BAR. What else could it be? I'm pretty stumped.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (GNU/Linux)

iD8DBQFEPELpX1gOwKyEAw8RAjT+AJ0ZzDb49tr5WwKWE7eyKWdT7hRLUQCgkOSS twDrsx8VrWG5xEf+hbbkFvg=
=im+D
-----END PGP SIGNATURE-----

#4 lore raw
In reply to Ian Romanick
Quoted 6 lines · Ian Romanick

This seems to mostly work, but I am having one problem. I map the region by opening the file with O_RDWR, then mmap with

(PROT_READ|PROT_WRITE) and MAP_SHARED.  In all cases, the open and mmap
succeed.  However, for I/O BARs, the resulting pointer from mmap is
invalid.  Any access to it results in a segfault and GDB says it's "out

of range".

On which architecture(s)?

Paul.

#5 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 10 lines · Paul Mackerras
Quoted 6 lines · Ian Romanick

This seems to mostly work, but I am having one problem. I map the region by opening the file with O_RDWR, then mmap with

(PROT_READ|PROT_WRITE) and MAP_SHARED.  In all cases, the open and mmap
succeed.  However, for I/O BARs, the resulting pointer from mmap is
invalid.  Any access to it results in a segfault and GDB says it's "out

of range".

On which architecture(s)?

I've only tried on x86-64 (an Athlon64 3000+ to be exact) so far. -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (GNU/Linux)

iD8DBQFEPRd4X1gOwKyEAw8RAhdxAKCX9Y/ju2jK7q5Wdzmb9kfwt9T3lwCbBwrq BIRJkEIGqSOrXtkDC3/2Fts=
=mePG
-----END PGP SIGNATURE-----

#2 lore raw
In reply to Ian Romanick
Quoted 25 lines · Ian Romanick
Quoted 11 lines · Ian Romanick

I'm in the process of modifying X to be civilized in it's handling of PCI devices on Linux. As part of that, I've modified it to map the /sys/bus/pci/device/*/resource[0-6] files instead of mucking about with /dev/mem.

This seems to mostly work, but I am having one problem. I map the region by opening the file with O_RDWR, then mmap with (PROT_READ|PROT_WRITE) and MAP_SHARED. In all cases, the open and mmap succeed. However, for I/O BARs, the resulting pointer from mmap is invalid. Any access to it results in a segfault and GDB says it's "out of range".

I was a little mistaken about this. The BAR that causes the problem is not I/O. It *is* memory.

01:00.0 VGA compatible controller: Matrox Graphics, Inc. G400/G450 (rev 03) (prog-if 00 [VGA])
Subsystem: Matrox Graphics, Inc. Millennium G400 16Mb SGRAM
Flags: bus master, medium devsel, latency 64, IRQ 11 Memory at cc000000 (32-bit, prefetchable) [size=32M] Memory at cfefc000 (32-bit, non-prefetchable) [size=16K] Memory at cf000000 (32-bit, non-prefetchable) [size=8M] Expansion ROM at cfee0000 [disabled] [size=64K]

Quoted 5 lines

When I open and mmap resource0 (the framebuffer) I get 0x2b9aa48ea000. When I open and mmap resource1 (the card's registers) I get 0x2b9aa68ea000. I can access the resource0 pointer all day long without problems. The firs access to the resource1 pointer results in a segfault.

Hm, that's strange. On my via machine I can access all my VGA resources correctly, but that's not x86-64. Maybe the x86-64 implementation of pci_mmap_page_range is doing the wrong thing for uncacheable, non-WC regions somehow?

Jesse

#3 lore raw
In reply to Ian Romanick
Quoted 12 lines · Ian Romanick

I was a little mistaken about this. The BAR that causes the problem is not I/O. It *is* memory.

01:00.0 VGA compatible controller: Matrox Graphics, Inc. G400/G450 (rev 03) (prog-if 00 [VGA])
Subsystem: Matrox Graphics, Inc. Millennium G400 16Mb SGRAM
Flags: bus master, medium devsel, latency 64, IRQ 11 Memory at cc000000 (32-bit, prefetchable) [size=32M] Memory at cfefc000 (32-bit, non-prefetchable) [size=16K] Memory at cf000000 (32-bit, non-prefetchable) [size=8M] Expansion ROM at cfee0000 [disabled] [size=64K]

Quoted 5 lines

When I open and mmap resource0 (the framebuffer) I get 0x2b9aa48ea000. When I open and mmap resource1 (the card's registers) I get 0x2b9aa68ea000. I can access the resource0 pointer all day long without problems. The firs access to the resource1 pointer results in a segfault.

Just tested on my x86-64 machine with this dumb little test program. It seems to work ok, though I haven't tried writing data to the resource.

Jesse

#include <errno.h>
#include <fcntl.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>

int main(int argc, char *argv[])

{
	size_t len;
	int fd, i;
	void *ptr;
	uint32_t *val;

	if (argc != 3) {
		fprintf(stderr, "usage: %s <file> <mapsize>\n",
			argv[0]);
		return -1;
	}

	len = atoi(argv[2]);

	fd = open(argv[1], O_RDONLY);
	if (fd == -1) {
		fprintf(stderr, "open failed: %s\n", strerror(errno));
		return errno;
	}

	ptr = mmap(NULL, len, PROT_READ, MAP_SHARED, fd, 0);
	if (ptr == MAP_FAILED) {
		fprintf(stderr, "mmap failed: %s\n", strerror(errno));
		return errno;
	}

	val = ptr;
	len = len / sizeof(uint32_t);
	for (i = 0; i < len; i += 4) {
		printf("%08x: 0x%08x 0x%08x 0x%08x 0x%08x\n", i * 4, val[i],
		       val[i+1], val[i+2], val[i+3]);
	}

	munmap(ptr, len); /* ignore any errors, we don't care */
	close(fd);
	return 0;
}