Skip to contentSkip to conversation

[Fwd: IPoIB]

Fwd: IPoIB Joseph Golio 2 participants 6 messages May 30, 2002 – Jun 6, 2002

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Joseph Golio wrote:

To whom it may concern,

I originally sent this note to Alan Cox, but he has suggested that I send it to you folks. I'm
new at this process so please bare with me...

So, here goes.

Quoted 29 lines

Alan,

I have a question. I am working on developing a Linux driver for IP over
Infiniband (IPoIB) and
have run into an issue that I need your advice. The draft standard from the
IETF on IPoIB
encapsulation and address resolution over Infiniband networks (see the link
below - section 6.1.1)
defines the hardware address as being 20 bytes in length. It appears that
the "netdevice.h" file in
Linux has MAX_ADDR_LEN set to 7 (at least in my version which is SuSe 7.3 -
kernel 2.4.4).

http://www.ietf.org/internet-drafts/draft-ietf-ipoib-ip-over-infiniband-01.txt

Obviously, this will not.

What do you suggest as a good solution to this problem ?

Thanks in advance for a quick response.

Thanks for your time,

Joe Golio
[email protected]
651-698-9350 (ext. 111)
VIEO Inc.

VIEO: The Adaptive Application Infrastructure Management (AAIM) Company

#1 lore raw

--snip/snip

Quoted 9 lines
Quoted 9 lines

I have a question. I am working on developing a Linux driver for IP over
Infiniband (IPoIB) and
have run into an issue that I need your advice. The draft standard from the
IETF on IPoIB
encapsulation and address resolution over Infiniband networks (see the link
below - section 6.1.1)
defines the hardware address as being 20 bytes in length. It appears that
the "netdevice.h" file in
Linux has MAX_ADDR_LEN set to 7 (at least in my version which is SuSe 7.3 -

for 2.5.18 at least it's set to 8, but there is no reason to not change it to 20 beside wasting some memory

n time for
a) broadcast address
b) device address
int netdevice
sum(m[n]) times for the multicast list

where n == number of network devices, m == number of MC entries per device as i can see it.

and this "overhead" should be really acceptable :)

... probably you will break some "external" stuff like freeswan, but this shouldn't be your problem.

tm

Signature · 1 line
in some way i do, and in some way i don't.
#2 lore raw

Thomas 'Dent' Mirlacher wrote:

That's what I thought. Now, I can make the change to my local systems for development and we could even give instructions to our customers to make the change to their systems also. However, how would I get this change propogated to the next release of the Linux kernel (2.5.18 plus...) ?

Thanks,

Joe

Quoted 33 lines

--snip/snip

Quoted 9 lines
Quoted 9 lines

I have a question. I am working on developing a Linux driver for IP over
Infiniband (IPoIB) and
have run into an issue that I need your advice. The draft standard from the
IETF on IPoIB
encapsulation and address resolution over Infiniband networks (see the link
below - section 6.1.1)
defines the hardware address as being 20 bytes in length. It appears that
the "netdevice.h" file in
Linux has MAX_ADDR_LEN set to 7 (at least in my version which is SuSe 7.3 -

for 2.5.18 at least it's set to 8, but there is no reason to not change it to 20 beside wasting some memory

n time for
a) broadcast address
b) device address
int netdevice
sum(m[n]) times for the multicast list

where n == number of network devices, m == number of MC entries per device as i can see it.

and this "overhead" should be really acceptable :)

... probably you will break some "external" stuff like freeswan, but this shouldn't be your problem.

tm

--
in some way i do, and in some way i don't.

#3 lore raw
In reply to Thomas 'Dent' Mirlacher

Thomas 'Dent' Mirlacher wrote:

Thomas,

Thanks for the reply. The more I dig into this, the more I figure out that the
problem is larger than I though. Not only do I need to increase the size of a hardware address, I also need to provide a new interface service routine to allow the device driver to lookup/translate a portion of the hardware address and return more information that needs to go into the ARP cache.

Specifically, the IETF IPoIB group has defined an Infiniband hardware address to have 1 byte of reserved, 3 bytes of Queue Pair Number (QPN) and 16 bytes of Global Identifier (GID). Unfortunately, in order to build the hardware header in packets to send, another piece of information is needed -- that being the Local Identifier (LID). If the LID is kept in the ARP cache, then it will be readily available
from "arp_find()" at the time of transmission. If we don't do that, we would have to have a separate table and mechanism in the driver to map from QPN/GID ===> LID. I would rather not do that if I didn't have to. It seems fundamental (at least to me) that
everything one needs to build the hardware header (at least everything that is related
specifically to the destination) should be stored in the ARP cache.

Thoughts anyone ?

Thanks,

Joe

Quoted 33 lines

--snip/snip

Quoted 9 lines
Quoted 9 lines

I have a question. I am working on developing a Linux driver for IP over
Infiniband (IPoIB) and
have run into an issue that I need your advice. The draft standard from the
IETF on IPoIB
encapsulation and address resolution over Infiniband networks (see the link
below - section 6.1.1)
defines the hardware address as being 20 bytes in length. It appears that
the "netdevice.h" file in
Linux has MAX_ADDR_LEN set to 7 (at least in my version which is SuSe 7.3 -

for 2.5.18 at least it's set to 8, but there is no reason to not change it to 20 beside wasting some memory

n time for
a) broadcast address
b) device address
int netdevice
sum(m[n]) times for the multicast list

where n == number of network devices, m == number of MC entries per device as i can see it.

and this "overhead" should be really acceptable :)

... probably you will break some "external" stuff like freeswan, but this shouldn't be your problem.

tm

--
in some way i do, and in some way i don't.

#4 lore raw

--snip/snip

Quoted 14 lines

The more I dig into this, the more I figure out that the
problem is larger than I though. Not only do I need to increase the size of a hardware address, I also need to provide a new interface service routine to allow the device driver to lookup/translate a portion of the hardware address and return more information that needs to go into the ARP cache.

Specifically, the IETF IPoIB group has defined an Infiniband hardware address to have 1 byte of reserved, 3 bytes of Queue Pair Number (QPN) and 16 bytes of Global Identifier (GID). Unfortunately, in order to build the hardware header in packets to send, another piece of information is needed -- that being the Local
Identifier (LID). If the LID is kept in the ARP cache, then it will be readily available from "arp_find()" at the time of transmission.

Quoted 7 lines

If we don't do that, we would have to
have a separate table and mechanism in the driver to map from QPN/GID ===> LID.
I would rather not do that if I didn't have to. It seems fundamental (at least to me) that everything one needs to build the hardware header (at least everything that is related
specifically to the destination) should be stored in the ARP cache.

you can already do that, since you will save everything in the arp cache which is the size of dev->addr_len. - did i miss the point?

cheers,

tm

Signature · 1 line
in some way i do, and in some way i don't.
#5 lore raw
Quoted 24 lines · Thomas 'Dent' Mirlacher

--snip/snip

Quoted 14 lines

The more I dig into this, the more I figure out that the
problem is larger than I though. Not only do I need to increase the size of a hardware address, I also need to provide a new interface service routine to allow the device driver to lookup/translate a portion of the hardware address and return more information that needs to go into the ARP cache.

Specifically, the IETF IPoIB group has defined an Infiniband hardware address to have 1 byte of reserved, 3 bytes of Queue Pair Number (QPN) and 16 bytes of Global Identifier (GID). Unfortunately, in order to build the hardware header in packets to send, another piece of information is needed -- that being the Local
Identifier (LID). If the LID is kept in the ARP cache, then it will be readily available from "arp_find()" at the time of transmission.

Quoted 7 lines

If we don't do that, we would have to
have a separate table and mechanism in the driver to map from QPN/GID ===> LID.
I would rather not do that if I didn't have to. It seems fundamental (at least to me) that everything one needs to build the hardware header (at least everything that is related
specifically to the destination) should be stored in the ARP cache.

Quoted 2 lines

you can already do that, since you will save everything in the arp cache which is the size of dev->addr_len. - did i miss the point?

Either you missed the point, I didn't make the point, or I'm missing something... :-)

Let me try this again. The IETF has defined the hardware address as having 2 fields.
The QPN and GID. Unfortunately, we need a 3rd item to build the headers. The LID. The LID can be gotten by IBTA methods once we know the GID. So, implementations that follow the IETF spec will be setting their local hardware address to their own

GID and some QPN. So, when I send out an ARP request and get back an ARP reply, it will have QPN and GID. So, what get's stored in my local ARP cache, if I don't do
anything to the ARP code, is simply the QPN/GID value. When I go to send a packet, I will call arp_find() and get back the QPN/GID. What I really need is the LID. SO,

either I would have to translate QPN/GID to LID at that time (NOT GOOD), or, back when the ARP reply came in, the ARP code could look at the arp_hrd and see if it's
(32) ARPHRD_INFINIBAND and if it is, call an interface service routine to
suplement
the ARP info before storing it in the cache. I know, it's not pretty, but I'm really struggling
here. If I were to do this in the driver, I would have to have some way of knowing that the
ARP had completed so that I could then go out and translate the GID to LID and stuff it
into a private driver lookup table (hashed probably). Not real crazy about that idea...

Joe

Quoted 8 lines

cheers,

tm

--
in some way i do, and in some way i don't.