Skip to contentSkip to conversation

PCI Express

DISCUSSION Colin Ngam 6 participants 32 messages Mar 4, 2005 – Mar 9, 2005

lore
Share & tools
lore

Colin Ngam

author
#0 lore raw

Hi,

Is MSI/MSIX support available for ia64?

Are PCIE devices supported in ia64?

A pointer to any documentations with regards to PCIE and ia64 will be very much appreciated.

Thanks.

colin

#1 lore raw
Quoted 1 line · Colin Ngam

Is MSI/MSIX support available for ia64?

Yes

Quoted 1 line

Are PCIE devices supported in ia64?

I'm having trouble with the word "supported". You have a PCI-E Bridge and firmware that can talk to it hooked up to an ia64 box?

Anyway, linux has some PCI-E support in the generic PCI code.

Quoted 2 lines

A pointer to any documentations with regards to PCIE and ia64 will be very much appreciated.

I don't think anything ia64 or linux specific exists.

grant

#2 lore raw
In reply to Colin Ngam
Quoted 8 lines · Grant Grundler
Quoted 1 line · Colin Ngam

Are PCIE devices supported in ia64?

I'm having trouble with the word "supported". You have a PCI-E Bridge and firmware that can talk to it hooked up to an ia64 box?

Anyway, linux has some PCI-E support in the generic PCI code.

... and I wrote the ia64-specific bits to access extended PCI config space through the SAL PCI accessors. See PCI_SAL_EXT_ADDRESS().

Various PCIe features are in the process of being added. It all depends what you mean when you say "supported".

Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
#3 lore raw
In reply to Colin Ngam
Quoted 8 lines · Matthew Wilcox
Quoted 7 lines · Grant Grundler
Quoted 1 line · Colin Ngam

Are PCIE devices supported in ia64?

I'm having trouble with the word "supported". You have a PCI-E Bridge and firmware that can talk to it hooked up to an ia64 box?

Quoted 8 lines
Quoted 1 line

Anyway, linux has some PCI-E support in the generic PCI code.

... and I wrote the ia64-specific bits to access extended PCI config space through the SAL PCI accessors. See PCI_SAL_EXT_ADDRESS().

Various PCIe features are in the process of being added. It all depends what you mean when you say "supported".

Are there specific lists (other than this one) to be watching for PCIe discussions?

thanks
Mark

#4 lore raw
In reply to Colin Ngam
Quoted 6 lines · Mark Maule
Quoted 2 lines · Matthew Wilcox

Various PCIe features are in the process of being added. It all depends what you mean when you say "supported".

Are there specific lists (other than this one) to be watching for PCIe discussions?

There's [email protected] and some stuff also gets discussed on l-k. There's not a whole lot of ia64-specific content, so I would only expect it to come up rarely on this list.

Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
author
#5 lore raw
In reply to Colin Ngam

Grant Grundler wrote:

Hi Grant,

Thanks for your respond.

Quoted 9 lines · Colin Ngam
Quoted 5 lines

Is MSI/MSIX support available for ia64?

Yes

I looked at Documentation/MSI-HOWTO.txt(linux-ia64-release-2.6.11) and it mentioned that CONFIG_X86_LOCAL_APIC has to be configured. I did not find this configured on in any sample ia64 config files, but it does exist in defconfig for i386. Is this just an ia32 config option?

Quoted 12 lines
Quoted 7 lines

Are PCIE devices supported in ia

I'm having trouble with the word "supported". You have a PCI-E Bridge and firmware that can talk to it hooked up to an ia64 box?

Pretty soon.

Is there any ia64 box with MSI and PCIE actually "activated and running"?

Thanks.

colin

Quoted 19 lines

Anyway, linux has some PCI-E support in the generic PCI code.

Quoted 4 lines

A pointer to any documentations with regards to PCIE and ia64 will be very much appreciated.

I don't think anything ia64 or linux specific exists.

grant
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#6 lore raw
In reply to Colin Ngam
Quoted 3 lines · Colin Ngam

I looked at Documentation/MSI-HOWTO.txt(linux-ia64-release-2.6.11) and

it mentioned that CONFIG_X86_LOCAL_APIC has to be configured. I did

not

Quoted 2 lines

find this configured on in any sample ia64 config files, but it does exist in defconfig for i386. Is this just an ia32 config option?

IA64 uses SAPIC instead of IOxAPIC. SAPIC, which is somewhat similar to IOxAPIC, is included in IA64 kernel. MSI/MSI-X support configuration option therefore is always available in IA64.

Thanks,
Long

#7 lore raw
In reply to Colin Ngam
Quoted 5 lines · Tom L
Quoted 3 lines · Colin Ngam

I looked at Documentation/MSI-HOWTO.txt(linux-ia64-release-2.6.11) and

it mentioned that CONFIG_X86_LOCAL_APIC has to be configured. I did

Quoted 8 lines

not

Quoted 2 lines

find this configured on in any sample ia64 config files, but it does exist in defconfig for i386. Is this just an ia32 config option?

IA64 uses SAPIC instead of IOxAPIC. SAPIC, which is somewhat similar to IOxAPIC, is included in IA64 kernel. MSI/MSI-X support configuration option therefore is always available in IA64.

So the MSIs are programmed to point at the processor SAPIC block? I think that'll work for us on Altix, but if they're pointed at an external Intel (or compatible) interrupt controller, we'll have to write new code for Altix.

Jesse

#8 lore raw
In reply to Colin Ngam
Quoted 4 lines · Colin Ngam

I looked at Documentation/MSI-HOWTO.txt(linux-ia64-release-2.6.11) and it mentioned that CONFIG_X86_LOCAL_APIC has to be configured. I did not find this configured on in any sample ia64 config files, but it does exist in defconfig for i386. Is this just an ia32 config option?

yes

Quoted 5 lines
Quoted 3 lines

I'm having trouble with the word "supported". You have a PCI-E Bridge and firmware that can talk to it hooked up to an ia64 box?

Pretty soon.
Is there any ia64 box with MSI and PCIE actually "activated and running"?

I've not seen anything published yet.

*Some* x86-64 boxes have working MSI (and PCI-E).
Recent PCI quirks have been submitted to flag the AMD chipsets that don't support MSI or MSI-X.

grant

#9 lore raw
In reply to Colin Ngam
Quoted 3 lines · Jesse Barnes
Quoted 2 lines · Tom L
Quoted 1 line · Colin Ngam

I looked at Documentation/MSI-HOWTO.txt(linux-ia64-release-2.6.11)

and

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

it mentioned that CONFIG_X86_LOCAL_APIC has to be configured. I

did

Quoted 4 lines
Quoted 4 lines

not

Quoted 1 line

find this configured on in any sample ia64 config files, but it

does

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

exist in defconfig for i386. Is this just an ia32 config option?

IA64 uses SAPIC instead of IOxAPIC. SAPIC, which is somewhat similar

to

Quoted 4 lines
Quoted 2 lines

IOxAPIC, is included in IA64 kernel. MSI/MSI-X support configuration option therefore is always available in IA64.

So the MSIs are programmed to point at the processor SAPIC block? I

think

Quoted 1 line

that'll work for us on Altix, but if they're pointed at an external

Intel >(or

Quoted 1 line

compatible) interrupt controller, we'll have to write new code for

Altix.

MSI/MSI-X message address is programmed to point at specific processor, not an external Intel interrupt controller, as a target.

Thanks,
Long

#10 lore raw
In reply to Colin Ngam
Quoted 3 lines · Jesse Barnes

So the MSIs are programmed to point at the processor SAPIC block? I think that'll work for us on Altix, but if they're pointed at an external Intel (or compatible) interrupt controller, we'll have to write new code for Altix.

Assuming your "Processor Interrupt Block" (IIRC) is at 0xfee00000 it should just work.

grant

#11 lore raw
In reply to Colin Ngam
Quoted 8 lines · Grant Grundler
Quoted 4 lines · Jesse Barnes

So the MSIs are programmed to point at the processor SAPIC block? I think that'll work for us on Altix, but if they're pointed at an external Intel (or compatible) interrupt controller, we'll have to write new code for Altix.

Assuming your "Processor Interrupt Block" (IIRC) is at 0xfee00000 it should just work.

Cool, I was hoping it just pointed at the same place we use for IPIs. It *should* work on Altix then, assuming the MSI allocation routines don't have platform knowledge in them (like ACPI bits or something).

Thanks,
Jesse

#12 lore raw
In reply to Colin Ngam
Quoted 3 lines · Jesse Barnes

Cool, I was hoping it just pointed at the same place we use for IPIs. It *should* work on Altix then, assuming the MSI allocation routines don't have platform knowledge in them (like ACPI bits or something).

I submitted a patch to add MSI support to tg3:

	ftp://ftp.parisc-linux.org/patches/diff-2.6.10-tg3_MSI-03

It was rejected because of HW bugs in bcm57xx chips. But it seems to work fine for me on the HP ZX1 platforms. Maybe it's worth trying on the altix boxes as well.

grant

author
#13 lore raw
In reply to Colin Ngam
Quoted 22 lines · Tom L
Quoted 3 lines · Jesse Barnes
Quoted 2 lines · Tom L
Quoted 1 line · Colin Ngam

I looked at Documentation/MSI-HOWTO.txt(linux-ia64-release-2.6.11)

and

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

it mentioned that CONFIG_X86_LOCAL_APIC has to be configured. I

did

Quoted 4 lines
Quoted 4 lines

not

Quoted 1 line

find this configured on in any sample ia64 config files, but it

does

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

exist in defconfig for i386. Is this just an ia32 config option?

IA64 uses SAPIC instead of IOxAPIC. SAPIC, which is somewhat similar

to

Quoted 4 lines
Quoted 2 lines

IOxAPIC, is included in IA64 kernel. MSI/MSI-X support configuration option therefore is always available in IA64.

So the MSIs are programmed to point at the processor SAPIC block? I

Quoted 8 lines

think

Quoted 1 line

that'll work for us on Altix, but if they're pointed at an external

Intel >(or

Quoted 1 line

compatible) interrupt controller, we'll have to write new code for

Altix.

MSI/MSI-X message address is programmed to point at specific processor, not an external Intel interrupt controller, as a target.

Hi Long/All,

Thank you (all) very much for the information.

It's been a long time since we tested MSI/MSIX on our Altix boxes. It has been longer since I looked at the code. It works and we are looking forward to hooking them up with the current Infrastructure available on ia64 - pci_enable|disable_msi().

On Altix systems, we have a set of "Interrupt Registers" in Memory Address Space that is initialized to target specific CPUs. The way we initialize a card's MSI is:

1. The Target Address is One of these "Interrupt Registers"
2. The Data Payload is the IRQ plus some special Altix bits.

This memory write causes the "Interrupt Chipset" to generate a LINTR message to the configured targeted cpu with the IRQ. Ofcourse, these registers are Altix Platform specific. Moreover, we have chunks of these registers all over the place.

Is there a more direct mechanism to generate an interrupt(LINTR Message) to a Processor? 1 of the Special bits that I mentioned in Item 2 above causes our hardware to flush all posted DMA buffers before allowing the LINTR Message to be generated to the cpu.

With Altix, it's always interesting, with respect to "just work" with typical ia64 code base. Hopefully, it will work cleaner compared to some of our other efforts.

Are you the maintainer for the MSI code?

Thanks.

colin

Quoted 4 lines

Thanks,
Long

#14 lore raw
In reply to Colin Ngam
Quoted 9 lines · Colin Ngam

It's been a long time since we tested MSI/MSIX on our Altix boxes. It has been longer since I looked at the code. It works and we are looking forward to hooking them up with the current Infrastructure available on ia64 - pci_enable|disable_msi().

On Altix systems, we have a set of "Interrupt Registers" in Memory Address Space that is initialized to target specific CPUs. The way we initialize a card's MSI is:

Quoted 7 lines

1. The Target Address is One of these "Interrupt Registers"
2. The Data Payload is the IRQ plus some special Altix bits.

This memory write causes the "Interrupt Chipset" to generate a LINTR message to the configured targeted cpu with the IRQ. Ofcourse, these registers are Altix Platform specific. Moreover, we have chunks of these registers all over the place.

That's one way to do it, but is obviously very Altix specific.

Quoted 4 lines

Is there a more direct mechanism to generate an interrupt(LINTR Message) to a Processor? 1 of the Special bits that I mentioned in Item 2 above causes our hardware to flush all posted DMA buffers before allowing the LINTR Message to be generated to the cpu.

Yes, see Grant's earlier message about the processor interrupt block. If we target MSIs at that, they'll behave the same way as IPIs, which are already platform independent (i.e. they work on Altix, zx1, tiger, etc.) since the processor interrupt block is builtin to the CPU. See Section 5.8.4 of Vol. 2 of the Itanium Arch. Software Developer's Manual (the blue books).

Jesse

#15 lore raw
In reply to Colin Ngam
Quoted 1 line · Colin

On Altix systems, we have a set of "Interrupt Registers" in Memory

Address

Quoted 9 lines

Space that is initialized to target specific CPUs. The way we initialize a card's MSI is:

1. The Target Address is One of these "Interrupt Registers"
2. The Data Payload is the IRQ plus some special Altix bits.

This memory write causes the "Interrupt Chipset" to generate a LINTR message to the configured targeted cpu with the IRQ. Ofcourse, these registers are Altix Platform specific. Moreover, we have chunks of

these

Quoted 1 line

registers all over the place.

Your solution is Altix chipset specific. MSI affects other architectures too. General solution would almost certainly be a better one.

Quoted 1 line

Is there a more direct mechanism to generate an interrupt(LINTR

Message)

Quoted 1 line

to a Processor?

I agree with Jesse's comment.

Quoted 5 lines

With Altix, it's always interesting, with respect to "just work" with typical ia64 code base. Hopefully, it will work cleaner compared to some of our other efforts.

Are you the maintainer for the MSI code?

I suggest you move any discussion or any proposed changes to LKML because MSI affects other architectures too. Again, general solution would almost certainly be a better one.

Thanks,
Long

author
#16 lore raw
In reply to Colin Ngam
Quoted 9 lines · Tom L
Quoted 1 line · Colin

On Altix systems, we have a set of "Interrupt Registers" in Memory

Address

Quoted 6 lines

Space that is initialized to target specific CPUs. The way we initialize a card's MSI is:

1. The Target Address is One of these "Interrupt Registers"
2. The Data Payload is the IRQ plus some special Altix bits.

Quoted 8 lines
Quoted 3 lines

This memory write causes the "Interrupt Chipset" to generate a LINTR message to the configured targeted cpu with the IRQ. Ofcourse, these registers are Altix Platform specific. Moreover, we have chunks of

these

Quoted 1 line

registers all over the place.

Your solution is Altix chipset specific. MSI affects other architectures too. General solution would almost certainly be a better one.

Hi All,

Yes it is and we are looking for a generic solution.

Quoted 7 lines
Quoted 3 lines

Is there a more direct mechanism to generate an interrupt(LINTR

Message)

Quoted 1 line

to a Processor?

I agree with Jesse's comment.

Well, unfortunately, we do not send IPI by using the Processor Interrupt Block. We actually
target a Special Altix Chipset "IPI Interrupt" register that ends up generating an IPI. That is why, we
have Platform Specific "send ipi" calls on ia64:

platform_send_ipi()

  File          Line
0 machvec.h     113 #define platform_send_ipi ia64_mv.send_ipi
1 machvec.h     282 #define platform_send_ipi ia64_send_ipi

2 machvec_sn2.h 86 #define platform_send_ipi sn2_send_IPI
3 machvec.h 95 #define platform_send_ipi ia64_mv.send_ipi
4 machvec.h 255 #define platform_send_ipi ia64_send_ipi
5 machvec_sn2.h 83 #define platform_send_ipi sn2_send_IPI

I do not know why we do not use the Processor Interrupt Block, but I will find out. But this does
not mean that we cannot use the PIB for MSI. Ofcourse, it may not be relocated at
0x0000 0000 FEE0 0000.

Quoted 11 lines
Quoted 7 lines

With Altix, it's always interesting, with respect to "just work" with typical ia64 code base. Hopefully, it will work cleaner compared to some of our other efforts.

Are you the maintainer for the MSI code?

I suggest you move any discussion or any proposed changes to LKML because MSI affects other architectures too. Again, general solution would almost certainly be a better one.

We will. We are looking for a general solution.

We will start looking through the MSI code see how best we can implement this functionality to support Altix platform. At that point we will post to lkml for guidance.

Thanks.

colin

Quoted 8 lines

Thanks,
Long
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#17 lore raw
In reply to Colin Ngam
Quoted 15 lines · Colin Ngam

Well, unfortunately, we do not send IPI by using the Processor Interrupt Block. We actually
target a Special Altix Chipset "IPI Interrupt" register that ends up generating an IPI. That is why, we
have Platform Specific "send ipi" calls on ia64:

platform_send_ipi()

  File          Line
0 machvec.h     113 #define platform_send_ipi ia64_mv.send_ipi
1 machvec.h     282 #define platform_send_ipi ia64_send_ipi

2 machvec_sn2.h 86 #define platform_send_ipi sn2_send_IPI
3 machvec.h 95 #define platform_send_ipi ia64_mv.send_ipi
4 machvec.h 255 #define platform_send_ipi ia64_send_ipi
5 machvec_sn2.h 83 #define platform_send_ipi sn2_send_IPI

Quoted 5 lines

I do not know why we do not use the Processor Interrupt Block, but I will find out. But this does
not mean that we cannot use the PIB for MSI. Ofcourse, it may not be relocated at
0x0000 0000 FEE0 0000.

That's beside the point though--MSIs don't use IPIs, they just use the processor interrupt block, which is also used by IPIs (on non-sn2 platforms). If the processor interrupt block works for us (pretty sure it does) MSIs targetted there should also work. The problem for us is if a processor tries to IPI a processor on a different node with the processor interrupt block, which I don't think MSIs will try to do.

Jesse

author
#18 lore raw
In reply to Colin Ngam
Quoted 16 lines · Jesse Barnes
Quoted 15 lines · Colin Ngam

Well, unfortunately, we do not send IPI by using the Processor Interrupt Block. We actually
target a Special Altix Chipset "IPI Interrupt" register that ends up generating an IPI. That is why, we
have Platform Specific "send ipi" calls on ia64:

platform_send_ipi()

  File          Line
0 machvec.h     113 #define platform_send_ipi ia64_mv.send_ipi
1 machvec.h     282 #define platform_send_ipi ia64_send_ipi

2 machvec_sn2.h 86 #define platform_send_ipi sn2_send_IPI
3 machvec.h 95 #define platform_send_ipi ia64_mv.send_ipi
4 machvec.h 255 #define platform_send_ipi ia64_send_ipi
5 machvec_sn2.h 83 #define platform_send_ipi sn2_send_IPI

Quoted 5 lines
Quoted 5 lines

I do not know why we do not use the Processor Interrupt Block, but I will find out. But this does
not mean that we cannot use the PIB for MSI. Ofcourse, it may not be relocated at
0x0000 0000 FEE0 0000.

Hi Jesse,

Quoted 5 lines

That's beside the point though--MSIs don't use IPIs, they just use the processor interrupt block, which is also used by IPIs (on non-sn2 platforms). If the processor interrupt block works for us (pretty sure it does) MSIs

That's what I will find out before deciding to use it. It is not used for IPIs on our platform, probably for very good reasons. If we can and the address is the same than it is a no brainer.

Quoted 4 lines

targetted there should also work. The problem for us is if a processor tries to IPI a processor on a different node with the processor interrupt block, which I don't think MSIs will try to do.

Perhaps you can elaborate. I am not sure what you are trying to say here.

Thanks.

colin

Quoted 3 lines

Jesse

author
#19 lore raw
In reply to Colin Ngam
Quoted 41 lines · Colin Ngam
Quoted 16 lines · Jesse Barnes
Quoted 15 lines · Colin Ngam

Well, unfortunately, we do not send IPI by using the Processor Interrupt Block. We actually
target a Special Altix Chipset "IPI Interrupt" register that ends up generating an IPI. That is why, we
have Platform Specific "send ipi" calls on ia64:

platform_send_ipi()

  File          Line
0 machvec.h     113 #define platform_send_ipi ia64_mv.send_ipi
1 machvec.h     282 #define platform_send_ipi ia64_send_ipi

2 machvec_sn2.h 86 #define platform_send_ipi sn2_send_IPI
3 machvec.h 95 #define platform_send_ipi ia64_mv.send_ipi
4 machvec.h 255 #define platform_send_ipi ia64_send_ipi
5 machvec_sn2.h 83 #define platform_send_ipi sn2_send_IPI

Quoted 5 lines
Quoted 5 lines

I do not know why we do not use the Processor Interrupt Block, but I will find out. But this does
not mean that we cannot use the PIB for MSI. Ofcourse, it may not be relocated at
0x0000 0000 FEE0 0000.

Hi Jesse,

Quoted 5 lines

That's beside the point though--MSIs don't use IPIs, they just use the processor interrupt block, which is also used by IPIs (on non-sn2 platforms). If the processor interrupt block works for us (pretty sure it does) MSIs

That's what I will find out before deciding to use it. It is not used for IPIs on our platform, probably for very good reasons. If we can and the address is the same than it is a no brainer.

Quoted 4 lines

targetted there should also work. The problem for us is if a processor tries to IPI a processor on a different node with the processor interrupt block, which I don't think MSIs will try to do.

Quoted 9 lines

Perhaps you can elaborate. I am not sure what you are trying to say here.

Thanks.

colin

Quoted 3 lines

Jesse

Hi Jesse,

I beleive the reason why we do not use the Processor Interrupt Block for generating interrupts is because it does not have the range to reach all the nodes that we can have in the system.

Thanks.

colin

Quoted 6 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#20 lore raw
In reply to Colin Ngam
Quoted 2 lines · Colin Ngam

That's what I will find out before deciding to use it. It is not used for IPIs on our platform, probably for very good reasons.

Yep, because IPIs generated from the local block to a distant processor won't work otherwise (afaik).

Quoted 8 lines

If we can and the
address is the same than it is a no brainer.

Quoted 3 lines

targetted there should also work. The problem for us is if a processor tries to IPI a processor on a different node with the processor interrupt block, which I don't think MSIs will try to do.

Perhaps you can elaborate. I am not sure what you are trying to say here.

The SHub manual is a little vague here, but my understanding is that in order to send remote IPIs we need to use the SHub, whereas local IPIs (i.e. processor to itself or to the other CPU on the same FSB) will work fine.

An MSI should behave like a processor sending an IPI to itself since its address can be targeted at the processor's interrupt block and set to generate a local interrupt. Is that right, Tom & Grant?

If this won't work for us, no biggie, we just have to abstract things a little more. We could make the MSI into a platform specific cookie that we can store a SHub/PIC/TIO address in and other platforms can use to target the processor interrupt block. Hopefully we won't have to do that though, since it would require a few more changes.

Jesse

author
#21 lore raw
In reply to Colin Ngam
Quoted 24 lines · Jesse Barnes
Quoted 2 lines · Colin Ngam

That's what I will find out before deciding to use it. It is not used for IPIs on our platform, probably for very good reasons.

Yep, because IPIs generated from the local block to a distant processor won't work otherwise (afaik).

Quoted 8 lines

If we can and the
address is the same than it is a no brainer.

Quoted 3 lines

targetted there should also work. The problem for us is if a processor tries to IPI a processor on a different node with the processor interrupt block, which I don't think MSIs will try to do.

Perhaps you can elaborate. I am not sure what you are trying to say here.

The SHub manual is a little vague here, but my understanding is that in order to send remote IPIs we need to use the SHub, whereas local IPIs (i.e. processor to itself or to the other CPU on the same FSB) will work fine.

An MSI should behave like a processor sending an IPI to itself since its address can be targeted at the processor's interrupt block and set to generate a local interrupt. Is that right, Tom & Grant?

Quoted 5 lines

If this won't work for us, no biggie, we just have to abstract things a little more. We could make the MSI into a platform specific cookie that we can store a SHub/PIC/TIO address in and other platforms can use to target the processor interrupt block. Hopefully we won't have to do that though, since it would require a few more changes.

May be the way we go. Needs more perusal of the MSI code.

Thanks.

colin

Quoted 7 lines

Jesse
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#22 lore raw
In reply to Colin Ngam

On Tue, Mar 08, 2005 at 05:29:13PM -0800, Jesse Barnes wrote:
...

Quoted 3 lines

An MSI should behave like a processor sending an IPI to itself since its address can be targeted at the processor's interrupt block and set to generate a local interrupt. Is that right, Tom & Grant?

Yes - that's how I understand it too.

Quoted 4 lines

If this won't work for us, no biggie, we just have to abstract things a little more. We could make the MSI into a platform specific cookie that we can store a SHub/PIC/TIO address in and other platforms can use to target the processor interrupt block.

Couple more options:
o add hooks for MSI quirk/fixup for *after* the MSI has been assigned by generic code.

o make sure all MSI only go to "node local" processors. I thought platform code has some control over the CPU EID and data portion of the MSI. But I haven't looked over that in a few monthes.

grant

author
#23 lore raw
In reply to Colin Ngam

Grant Grundler wrote:

Hi Grant,

Thanks for the info.

Bottomline - Processor Interrupt Block not implemented on Altix.

Thanks.

colin

Quoted 26 lines · Jesse Barnes

...

Quoted 3 lines

An MSI should behave like a processor sending an IPI to itself since its address can be targeted at the processor's interrupt block and set to generate a local interrupt. Is that right, Tom & Grant?

Yes - that's how I understand it too.

Quoted 4 lines

If this won't work for us, no biggie, we just have to abstract things a little more. We could make the MSI into a platform specific cookie that we can store a SHub/PIC/TIO address in and other platforms can use to target the processor interrupt block.

Couple more options:
o add hooks for MSI quirk/fixup for *after* the MSI has been assigned by generic code.

o make sure all MSI only go to "node local" processors. I thought platform code has some control over the CPU EID and data portion of the MSI. But I haven't looked over that in a few monthes.

grant
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#24 lore raw
In reply to Colin Ngam

On Tue, Mar 08, 2005 at 05:29:13PM -0800, Jesse Barnes wrote:
...

Quoted 1 line

An MSI should behave like a processor sending an IPI to itself since

its

Quoted 2 lines

address can be targeted at the processor's interrupt block and set to generate a local interrupt. Is that right, Tom & Grant?

MSI address is programmed at 0xfeex_xxxx (based on PCI Specs) to generate a memory write to FSB. This address can be configured to target a platform interrupt controller (like IOAPIC for example), which in turn generates a memory write to a FSB, or be configured to send a memory write directly to FSB. Existing MSI support implements a direct memory-write mechanism (generic solution without platform dependency and limitation to MSI-X support as an example). With existing direct memory-write mechanism, a device's MSI address is configured with current running CPU as a target. MSI support also implements MSI SMP affinity, which supports IRQ rebalance and allows user to select which CPU target through /proc/irq/...

I hope it helps.

Thanks,
Long

#25 lore raw
In reply to Colin Ngam
Quoted 5 lines · Tom L

Existing MSI support implements a direct
memory-write mechanism (generic solution without platform dependency and limitation to MSI-X support as an example). With existing direct memory-write mechanism, a device's MSI address is configured with current running CPU as a target.

At some point, PARISC (and Alpha?) will also want to support MSI using "direct memory write". But like Altix, does not implement 0xfeeXXXXX (1) PIB since PARISC arch predates PCI by a decade or so.

We'll have to figure out which platform specific hooks would enable parisc/alpha/altix to use existing PCI MSI/MSI-X support.

PA-RISC IPI is as straight forward as Jesse described - just write to a per CPU address with a data value (vector).

thanks,
grant

(1) the most recent two or three PA-RISC chipsets do implement 0xfeeXXXXX interrupt block. But firmware hands us the CPU "ID/EID" values to use when programming the IO SAPIC IRT. See drivers/parisc/iosapic.c iosapic_set_irt_data() for more details.

Bottomline is, even if we know it's implemented in HW, we don't know if we are using 0xfeeXXXXX or not for IO SAPIC today. platform hooks into MSI code would have to sort that out.

author
#26 lore raw
In reply to Colin Ngam
Quoted 11 lines · Grant Grundler
Quoted 7 lines · Tom L

Existing MSI support implements a direct
memory-write mechanism (generic solution without platform dependency and limitation to MSI-X support as an example). With existing direct memory-write mechanism, a device's MSI address is configured with current running CPU as a target.

Quoted 7 lines

At some point, PARISC (and Alpha?) will also want to support MSI using "direct memory write". But like Altix, does not implement 0xfeeXXXXX (1) PIB since PARISC arch predates PCI by a decade or so.

We'll have to figure out which platform specific hooks would enable parisc/alpha/altix to use existing PCI MSI/MSI-X support.

Hi Grant,

Sounds great.

Thanks.

colin

Quoted 17 lines

PA-RISC IPI is as straight forward as Jesse described - just write to a per CPU address with a data value (vector).

thanks,
grant

(1) the most recent two or three PA-RISC chipsets do implement 0xfeeXXXXX interrupt block. But firmware hands us the CPU "ID/EID" values to use when programming the IO SAPIC IRT. See drivers/parisc/iosapic.c iosapic_set_irt_data() for more details.

Bottomline is, even if we know it's implemented in HW, we don't know if we are using 0xfeeXXXXX or not for IO SAPIC today. platform hooks into MSI code would have to sort that out.

author
#28 lore raw
In reply to Colin Ngam
Quoted 10 lines · Tom L

On Tue, Mar 08, 2005 at 05:29:13PM -0800, Jesse Barnes wrote:
...

Quoted 3 lines

An MSI should behave like a processor sending an IPI to itself since

its

Quoted 7 lines
Quoted 4 lines

address can be targeted at the processor's interrupt block and set to generate a local interrupt. Is that right, Tom & Grant?

MSI address is programmed at 0xfeex_xxxx (based on PCI Specs) to

Hi Tom,

The MSI address portion allows 64bits for the address. It should be an address whereby a specific Platform has configured with sufficient information to generate an Interrupt to the targeted "cpu" with the payload - IRQ.

I am very much interested in your assertion that the MSI address is defined by PCI Spec to be 0xfeex_xxxx. Can you kindly point me to the relevent section? Perhaps if you have an online copy, can you cut and paste the relevent section?

Thanks.

colin

Quoted 17 lines

generate a memory write to FSB. This address can be configured to target a platform interrupt controller (like IOAPIC for example), which in turn generates a memory write to a FSB, or be configured to send a memory write directly to FSB. Existing MSI support implements a direct memory-write mechanism (generic solution without platform dependency and limitation to MSI-X support as an example). With existing direct memory-write mechanism, a device's MSI address is configured with current running CPU as a target. MSI support also implements MSI SMP affinity, which supports IRQ rebalance and allows user to select which CPU target through /proc/irq/...

I hope it helps.

Thanks,
Long

#29 lore raw
In reply to Colin Ngam
Quoted 1 line · Colin Ngam

The MSI address portion allows 64bits for the address.

Yes - *allows*. It also allows devices to implement 32-bit MSI address. MSI "Control" bits indicate what is implemented.
See PCI2.2 spec or later and search for MSI.

Quoted 2 lines

I am very much interested in your assertion that the MSI address is defined by PCI Spec to be 0xfeex_xxxx.

...

Quoted 1 line

Can you kindly point me to the relevent section?

It's not and it doesn't belong there.

IA64 specs defines Processor Interrupt Block and where it lives. See "Intel® Itanium® Software Developer's Manual"

	Volume 2, Part 1, Section 5.8.4
	Volume 2, Part 1, Section 11.9.3

grant

#30 lore raw
In reply to Colin Ngam
Quoted 11 lines · Grant Grundler
Quoted 2 lines

I am very much interested in your assertion that the MSI address is defined by PCI Spec to be 0xfeex_xxxx.

...

Quoted 1 line

Can you kindly point me to the relevent section?

It's not and it doesn't belong there.

IA64 specs defines Processor Interrupt Block and where it lives. See "Intel(r) Itanium(r) Software Developer's Manual"

	Volume 2, Part 1, Section 5.8.4
	Volume 2, Part 1, Section 11.9.3

I apologize that it is not in PCI Spec. It is also in Section 8.11 of "IA-32 Intel Architecture Software Developer's Manual (volume 3)."

Thanks,
Long

author
#31 lore raw
In reply to Colin Ngam
Quoted 23 lines · Tom L
Quoted 10 lines · Grant Grundler
Quoted 4 lines

I am very much interested in your assertion that the MSI address is defined by PCI Spec to be 0xfeex_xxxx.

...

Quoted 3 lines

Can you kindly point me to the relevent section?

Quoted 8 lines

It's not and it doesn't belong there.

IA64 specs defines Processor Interrupt Block and where it lives. See "Intel(r) Itanium(r) Software Developer's Manual"

	Volume 2, Part 1, Section 5.8.4
	Volume 2, Part 1, Section 11.9.3
Quoted 3 lines

I apologize that it is not in PCI Spec. It is also in Section 8.11 of "IA-32 Intel Architecture Software Developer's Manual (volume 3)."

Hi,

Thanks Tom.

I believe we are all are in agreement that the MSI Address needs to be Platform "Tunable" :-)

Thank you.

colin

Quoted 9 lines

Thanks,
Long
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html