Skip to contentSkip to conversation

clean way to support >32bit addr on 32bit CPU

DISCUSSION Dave 7 participants 13 messages Jan 10, 2005 – Jan 11, 2005

lore
Share & tools
lore

Dave

author
#0 lore raw

I have this ARM (XScale) based platform that supports 36bit physical addressing. Due to the way the ATU is designed, the outbound memory translation window is fixed outside the first 4GB of memory space, and thus the need to use 64bit addressing in order to access the PCI bus. After all said and done, the struct resource members start and end must support 64bit integer values in order to work. On a 64bit arch that would be fine since unsigned long is 64bit. However on a 32bit arch one must use unsigned long long to get 64bit. However, if we do that then it would make the 64bit archs to have 128bit start and end and probably wouldn't be something we'd want. What would be a nice clean way to support this that's acceptable to Linux? I guess this issue would be similar to x86-32 PAE would have?

Also, please cc me on on the discussion. Not sure if my LKML subscription is working... Thanks!

Signature · 11 lines
-= Dave =-

Software Engineer - Advanced Development Engineering Team 
Storage Component Division - Intel Corp. 
mailto://dave.jiang @ intel dot com
http://sourceforge.net/projects/xscaleiop/
----
The views expressed in this email are
mine alone and do not necessarily 
reflect the views of my employer
(Intel Corp.).
#1 lore raw
Quoted 12 lines · Dave

I have this ARM (XScale) based platform that supports 36bit physical
addressing. Due to the way the ATU is designed, the outbound memory
translation window is fixed outside the first 4GB of memory space, and
thus the need to use 64bit addressing in order to access the PCI bus. After all said and done, the struct resource members start and end must support 64bit integer values in order to work. On a 64bit arch
that would be fine since unsigned long is 64bit. However on a 32bit
arch one must use unsigned long long to get 64bit. However, if we do
that then it would make the 64bit archs to have 128bit start and end
and probably wouldn't be something we'd want. What would be a nice
clean way to support this that's acceptable to Linux? I guess this
issue would be similar to x86-32 PAE would have?

Quoted 5 lines

Also, please cc me on on the discussion. Not sure if my LKML subscription is working... Thanks!

Also, it would be nice to have PTEs to represent the upper 4GB such that it can be mmapped to user space. PAE handled this in and it would be great to have it in ARM MMU36 as well.

-Slade

#2 lore raw

On Jan 10 2005, at 16:01, Slade Maurer was caught saying:

Quoted 3 lines

Also, it would be nice to have PTEs to represent the upper 4GB such that it can be mmapped to user space. PAE handled this in and it would be great to have it in ARM MMU36 as well.

Not doable. I believe PAE allows for normal 4K pages to be used when mapping > 32-bits. XSC3 and ARMv6 only allow for > 32 bit addresses when using 16MB pages (supersections), so we need to instead use the hugetlb approach.

~Deepak

Signature · 3 lines
Deepak Saxena - [email protected] - http://www.plexity.net

If you complain once more, you'll meet an army of me. - Bjork
#3 lore raw
Quoted 7 lines · Deepak Saxena

On Jan 10 2005, at 16:01, Slade Maurer was caught saying:

Quoted 3 lines

Also, it would be nice to have PTEs to represent the upper 4GB such that it can be mmapped to user space. PAE handled this in and it would be great to have it in ARM MMU36 as well.

Quoted 8 lines

Not doable. I believe PAE allows for normal 4K pages to be used when
mapping > 32-bits. XSC3 and ARMv6 only allow for > 32 bit addresses when using 16MB pages (supersections), so we need to instead use the hugetlb approach.

~Deepak

You are right of course. The MMUs first level descriptors force you to have 16MB pages.

I don't see anything wrong with using hugeTLB. Then it is up to the user to get hugetlbfs setup so that they can mmap(...) properly. This is forced on us by the designers of the MMU ;)

I think that is better than setting permissions during ioremap(...) so that a user space process can use a kernel virtual address for user space access.

-Slade

#4 lore raw
In reply to Dave
Quoted 5 lines · Dave

After all said and done, the struct resource members start and end must support 64bit integer values in order to work. On a 64bit arch that would be fine since unsigned long is 64bit. However on a 32bit arch one must use unsigned long long to get 64bit.

We really should make "struct resource" use u64's. It's wrong even on x86, but we've never seen any real problems in practice, so we've had little reason to bother.

This has definitely come up before, maybe there's even some old patch floating around. It should be as easy as just fixing up "start/end" to be "u64" (and perhaps move them to the beginning of the struct to make sure packing is ok on all architectures), and fixing any fall-out.

Linus

#5 lore raw
Quoted 9 lines · Linus Torvalds
Quoted 4 lines · Dave

After all said and done, the struct resource members start and end must support 64bit integer values in order to work. On a 64bit arch that would be fine since unsigned long is 64bit. However on a 32bit arch one must use unsigned long long to get 64bit.

Quoted 3 lines

We really should make "struct resource" use u64's. It's wrong even on x86, but we've never seen any real problems in practice, so we've had little reason to bother.

This has definitely come up before, maybe there's even some old patch floating around. It should be as easy as just fixing up "start/end" to be "u64" (and perhaps move them to the beginning of the struct to make sure packing is ok on all architectures), and fixing any fall-out.

Speaking of fall-out, or more like trickle-down, I'm almost done with a patch to make PCMCIA resources use unsigned long instead of u_int or u_short for IO address:

incluce/pcmcia/cs_types.h:
#if defined(__arm__) || defined(__mips__)
typedef u_int ioaddr_t;
#else
typedef u_short ioaddr_t;
#endif

becomes:
typedef unsigned long ioaddr_t;

and then include/pcmcia/cs.c needs some changes in use of ioaddr_t, along with drivers (printk formats).

Does that sound OK?
I guess that it would become unsigned long long (or u64) with this proposal?

Signature · 1 line
~Randy
#6 lore raw
Quoted 4 lines · Randy.Dunlap

Speaking of fall-out, or more like trickle-down, I'm almost done with a patch to make PCMCIA resources use unsigned long instead of u_int or u_short for IO address:

Ahh, yes. That's required on pretty much all platforms except x86 and x86-64.

Of course, since ARM and MIPS already do the "u_int" thing, and not a whole lot of other architectures do PCMCIA, I guess it doesn't matter _that_ much. Cardbus stuff should get it right regardless.

Quoted 8 lines

typedef unsigned long ioaddr_t;

and then include/pcmcia/cs.c needs some changes in use of ioaddr_t, along with drivers (printk formats).

Does that sound OK?
I guess that it would become unsigned long long (or u64) with this proposal?

I don't think ioaddr_t needs to match resources. None of the IO accessor functions take "u64"s anyway - and aren't likely to do so in the future either - so "unsigned long" should be good enough.

Having u64 for resource handling is mainly an issue for RAM and memory-mapped IO (right now the 32-bit limit means that we throw away information about stuff above the 4GB mark from the e820 interfaces on x86, for example - that _happens_ to work because we never see anything but RAM there anyway, but it means that /proc/iomem doesn't show all of the system RAM, and it does mean that our resource management doesn't actually handle 64-bit addresses correctly.

See drivers/pci/probe.c for the result:

"PCI: Unable to handle 64-bit address for device xxxx"

(and I do not actually think this has _ever_ happened in real life, which makes me suspect that Windows doesn't handle them either - but it inevitably will happen some day).

Linus

#7 lore raw
Quoted 7 lines · Linus Torvalds

I don't think ioaddr_t needs to match resources. None of the IO accessor functions take "u64"s anyway - and aren't likely to do so in the future either - so "unsigned long" should be good enough.
Having u64 for resource handling is mainly an issue for RAM and memory-mapped IO (right now the 32-bit limit means that we throw away information about stuff above the 4GB mark from the e820 interfaces on x86, for example - that _happens_ to work because we never see anything

Quoted 8 lines

but RAM there anyway, but it means that /proc/iomem doesn't show all of the system RAM, and it does mean that our resource management doesn't actually handle 64-bit addresses correctly.
See drivers/pci/probe.c for the result:
"PCI: Unable to handle 64-bit address for device xxxx"
(and I do not actually think this has _ever_ happened in real life, which makes me suspect that Windows doesn't handle them either - but it inevitably will happen some day).

I have a vague recollection of seeing a report of an ia32 device and/or machine with this property from John Fusco but am having a tough time searching the archives properly for it. I do recall it being around the time the remap_pfn_range() work was started, and I also claimed it as one of the motivators of it in one of my posts. I'm unaware of whether there are more general resources in John Fusco's situation.

My follow-ups began with:
Message-ID: <[email protected]>
References: <[email protected]>

-- wli

#8 lore raw
Quoted 20 lines · William Lee Irwin III
Quoted 9 lines · Linus Torvalds

I don't think ioaddr_t needs to match resources. None of the IO accessor
functions take "u64"s anyway - and aren't likely to do so in the future
either - so "unsigned long" should be good enough.
Having u64 for resource handling is mainly an issue for RAM and
memory-mapped IO (right now the 32-bit limit means that we throw away
information about stuff above the 4GB mark from the e820 interfaces on
x86, for example - that _happens_ to work because we never see anything but RAM there anyway, but it means that /proc/iomem doesn't show all of the system RAM, and it does mean that our resource management doesn't actually handle 64-bit addresses correctly. See drivers/pci/probe.c for the result:
"PCI: Unable to handle 64-bit address for device xxxx"
(and I do not actually think this has _ever_ happened in real life, which makes me suspect that Windows doesn't handle them either - but it inevitably will happen some day).

I have a vague recollection of seeing a report of an ia32 device and/or machine with this property from John Fusco but am having a tough time searching the archives properly for it. I do recall it being around the time the remap_pfn_range() work was started, and I also claimed it as one of the motivators of it in one of my posts. I'm unaware of whether there are more general resources in John Fusco's situation.

Quoted 3 lines

My follow-ups began with:
Message-ID: <[email protected]>
References: <[email protected]>

http://marc.theaimsgroup.com/?l=linux-mm&m=109598180125156&w=2

Signature · 1 line
~Randy
#9 lore raw
In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Randy.Dunlap

Speaking of fall-out, or more like trickle-down, I'm almost done with a patch to make PCMCIA resources use unsigned long instead of u_int or u_short for IO address:

Ahh, yes. That's required on pretty much all platforms except x86 and x86-64.

OK, I don't get it, sorry. What's different about ARM & MIPS here (for PCMCIA)? Is this historical (so that I'm just missing it) or is it a data types difference?

Quoted 8 lines

Of course, since ARM and MIPS already do the "u_int" thing, and not a whole lot of other architectures do PCMCIA, I guess it doesn't matter _that_ much. Cardbus stuff should get it right regardless.

Quoted 5 lines

typedef unsigned long ioaddr_t;

and then include/pcmcia/cs.c needs some changes in use of ioaddr_t, along with drivers (printk formats).

Quoted 8 lines
Quoted 3 lines

Does that sound OK?
I guess that it would become unsigned long long (or u64) with this proposal?

I don't think ioaddr_t needs to match resources. None of the IO accessor functions take "u64"s anyway - and aren't likely to do so in the future either - so "unsigned long" should be good enough.

Thanks,

Signature · 1 line
~Randy
#10 lore raw
Quoted 7 lines · Randy.Dunlap
Quoted 3 lines

Ahh, yes. That's required on pretty much all platforms except x86 and x86-64.

OK, I don't get it, sorry. What's different about ARM & MIPS here (for PCMCIA)? Is this historical (so that I'm just missing it) or is it a data types difference?

Nothing is different. Pretty much every architecture - except for x86 and ilk - will at least have the _potential_ for IO ports encoded above the 16-bit mark.

But a lot of architectures won't have PCMCIA (or if they do, they end up having the whole ISA mapping, and for compatibility reasons they'll end up making the ports visible to the kernel in the low 16 bits, even if the actual hw has some other physical translation - I think that's true on ppc, at least).

So what makes ARM and MIPS special is just the fact that they have PCMCIA, but don't necessarily have the traditional ISA mappings. Embedded devices and all that. Others either try to hide the fact that they look different, or just never cared.

But the right thing is definitely to make an IO port pointer be "unsigned int" or even "unsigned long".

Linus

author
#11 lore raw
In reply to Linus Torvalds
Quoted 13 lines · Linus Torvalds
Quoted 5 lines · Dave

After all said and done, the struct resource members start and end must support 64bit integer values in order to work. On a 64bit arch that would be fine since unsigned long is 64bit. However on a 32bit arch one must use unsigned long long to get 64bit.

We really should make "struct resource" use u64's. It's wrong even on x86, but we've never seen any real problems in practice, so we've had little reason to bother.

Quoted 7 lines

This has definitely come up before, maybe there's even some old patch floating around. It should be as easy as just fixing up "start/end" to be "u64" (and perhaps move them to the beginning of the struct to make sure packing is ok on all architectures), and fixing any fall-out.

Linus

Shall I change the PCI resource stuff also to be u64 or leave that alone? Currently I think we assume on 32bit processors all PCI resources are in the first 4GB region. Just wondering if that should be left alone....

Signature · 11 lines
-= Dave =-

Software Engineer - Advanced Development Engineering Team 
Storage Component Division - Intel Corp. 
mailto://dave.jiang @ intel
http://sourceforge.net/projects/xscaleiop/
----
The views expressed in this email are
mine alone and do not necessarily 
reflect the views of my employer
(Intel Corp.).
#12 lore raw
In reply to Dave

Dave> I have this ARM (XScale) based platform that supports 36bit Dave> physical addressing. Due to the way the ATU is designed, the Dave> outbound memory translation window is fixed outside the Dave> first 4GB of memory space, and thus the need to use 64bit Dave> addressing in order to access the PCI bus. After all said Dave> and done, the struct resource members start and end must Dave> support 64bit integer values in order to work. On a 64bit Dave> arch that would be fine since unsigned long is Dave> 64bit. However on a 32bit arch one must use unsigned long Dave> long to get 64bit. However, if we do that then it would make Dave> the 64bit archs to have 128bit start and end and probably Dave> wouldn't be something we'd want. What would be a nice clean Dave> way to support this that's acceptable to Linux? I guess this Dave> issue would be similar to x86-32 PAE would have?

Actually unsigned long long is still 64 bits even on 64-bit Linux architectures. However it might make more sense to use an explicit size for the resource addresses, namely u64.

This XScale architecture seems similar to the PowerPC 440, which also uses 36-bit addressing for various buses including PCI. You might want to take a look at arch/ppc for inspiration.

- Roland