ipv6: alternative version of S/390 shared NIC support

Let's try again solvign the EUI64 generation for S/390. I looked over the IBM patch and I think it can be done a lot simpler:

- put a dev_id field in struct net_device, so that it uses space that would be wasted by padding otherwise.
- if this fields is non-null let ipv6_generate_eui64 use the algorithm from the QETH code to generate an EUI that's different for each OS instance. See code comments for details.

drivers/s390/net/qeth_main.c +1 -25

@@ -5033,27 +5033,6 @@
 	return 0;
 }
 
-#ifdef CONFIG_QETH_IPV6
-int
-qeth_ipv6_generate_eui64(u8 * eui, struct net_device *dev)
-{
-	switch (dev->type) {
-	case ARPHRD_ETHER:
-	case ARPHRD_FDDI:
-	case ARPHRD_IEEE802_TR:
-		if (dev->addr_len != ETH_ALEN)
-			return -1;
-		memcpy(eui, dev->dev_addr, 3);
-		memcpy(eui + 5, dev->dev_addr + 3, 3);
-		eui[3] = (dev->dev_id >> 8) & 0xff;
-		eui[4] = dev->dev_id & 0xff;
-		return 0;
-	}
-	return -1;
-
-}
-#endif
-
 static void
 qeth_get_mac_for_ipm(__u32 ipm, char *mac, struct net_device *dev)
 {

@@ -5587,11 +5566,8 @@
 	}
 #ifdef CONFIG_QETH_IPV6
 	/*IPv6 address autoconfiguration stuff*/
-	card->dev->dev_id = card->info.unique_id & 0xffff;
 	if (!(card->info.unique_id & UNIQUE_ID_NOT_BY_CARD))
-		card->dev->generate_eui64 = qeth_ipv6_generate_eui64;
-
-
+		card->dev->dev_id = card->info.unique_id & 0xffff;
 #endif
 	dev->hard_header_parse = NULL;
 	dev->set_mac_address = qeth_layer2_set_mac_address;

include/linux/netdevice.h +1 -0

@@ -345,6 +345,7 @@
 	unsigned char		broadcast[MAX_ADDR_LEN];	/* hw bcast add	*/
 	unsigned char		dev_addr[MAX_ADDR_LEN];	/* hw address	*/
 	unsigned char		addr_len;	/* hardware address length	*/
+	unsigned short          dev_id;		/* for shared network cards */
 
 	struct dev_mc_list	*mc_list;	/* Multicast mac addresses	*/
 	int			mc_count;	/* Number of installed mcasts	*/

net/ipv6/addrconf.c +23 -4

@@ -1079,10 +1079,29 @@
 		if (dev->addr_len != ETH_ALEN)
 			return -1;
 		memcpy(eui, dev->dev_addr, 3);
-		memcpy(eui + 5, dev->dev_addr+3, 3);
-		eui[3] = 0xFF;
-		eui[4] = 0xFE;
-		eui[0] ^= 2;
+		memcpy(eui + 5, dev->dev_addr + 3, 3);
+
+		/*
+		 * The zSeries OSA network cards can be shared among various
+		 * OS instances, but the OSA cards have only one MAC address.
+		 * This leads to duplicate address conflicts in conjunction
+		 * with IPv6 if more than one instance uses the same card.
+		 * 
+		 * The driver for these cards can deliver a unique 16-bit
+		 * identifier for each instance sharing the same card.  It is
+		 * placed instead of 0xFFFE in the interface identifier.  The
+		 * "u" bit of the interface identifier is not inverted in this
+		 * case.  Hence the resulting interface identifier has local
+		 * scope according to RFC2373.
+		 */
+		if (dev->dev_id) {
+			eui[3] = (dev->dev_id >> 8) & 0xFF;
+			eui[4] = dev->dev_id & 0xFF;
+		} else {
+			eui[3] = 0xFF;
+			eui[4] = 0xFE;
+			eui[0] ^= 2;
+		}
 		return 0;
 	case ARPHRD_ARCNET:
 		/* XXX: inherit EUI-64 from other interface -- yoshfuji */

I could be missing something:
Its almost like the "cards" in these drivers deserve to be their own netdevices to begin with. In other words the qeth approach itself seems to be a hack to begin with.
Should be along the lines:
--->Physical netdevice
-->some demux code to select instance

    --> Virtual netdevice specific to OS instance
        ---> standard linux path

The problem is again that the OS instances don't talk to each (well they can, but not in the driver), so your demux code would have to move into the device firmware, and that'd probably change the device <-> OS interface completely.

Quoted 4 lines · Christoph Hellwig

The problem is again that the OS instances don't talk to each (well they can, but not in the driver), so your demux code would have to move into the device firmware, and that'd probably change the device <-> OS interface completely.

qeth_qdio_input_handler() seems to know what "card" the packet came on, no? Could you not use this information to select the proper netdevice? Actually, how does that work? Each OS instance would get the same driver invoked but you will never see any othe instances "cards"?

I am trying a small simplification here:
Each physical network adapter offers hundreds of device addresses. You need 3 of them to have one logical network adapter(read,write,data).

S/390 has
hardware supported virtualization. Therefore can then use the hypervisor (LPAR or z/VM) to give specific LPARs or VM guests exactly 3 device addresses out of these hundreds.

The qeth driver has to register the IP address at the logical network card (using 3 device addresses) Afterwards the physical network card knows which packet belongs to which device numbers.

Quoted 3 lines · Christian Bornträger

I am trying a small simplification here:
Each physical network adapter offers hundreds of device addresses. You need 3 of them to have one logical network adapter(read,write,data).

the "card" concept is what you call network adapter, correct? I take it that read and write are control channels and data is where the skb comes through?

Quoted 4 lines

S/390 has
hardware supported virtualization. Therefore can then use the hypervisor (LPAR or z/VM) to give specific LPARs or VM guests exactly 3 device addresses out of these hundreds.

Can you provision multiple of these cards per VM? if yes, is there some ID that will break it down to OSInstance:cardid?

Quoted 3 lines

The qeth driver has to register the IP address at the logical network card (using 3 device addresses) Afterwards the physical network card knows which packet belongs to which device numbers.

I think i understood but confused: before you attach IP address though, you cant receive packets? Is there a concept of MAC address which you can pass to the hypervisor or can you run in promiscmous mode?

Another question: When that driver runs for the physical card - it runs

in the context of a specific VM, correct? In other words it would be impossible to see the "card" of another instance?

Frank, please correct me, if I am wrong....

Quoted 3 lines · jamal

Quoted 2 lines · Christian Bornträger

I am trying a small simplification here:
Each physical network adapter offers hundreds of device addresses.

Quoted 4 lines

Quoted 1 line

need 3 of them to have one logical network adapter(read,write,data).

the "card" concept is what you call network adapter, correct? I take it that read and write are control channels and data is where

Quoted 1 line

skb comes through?

don't ask me about naming....

Quoted 3 lines

Quoted 3 lines

S/390 has
hardware supported virtualization. Therefore can then use the hypervisor (LPAR or z/VM) to give specific LPARs or VM guests exactly

Quoted 9 lines

Quoted 1 line

device addresses out of these hundreds.

Can you provision multiple of these cards per VM? if yes, is there some ID that will break it down to OSInstance:cardid?

Quoted 3 lines

The qeth driver has to register the IP address at the logical network card (using 3 device addresses) Afterwards the physical network card knows which packet belongs to which device numbers.

I think i understood but confused: before you attach IP address

Quoted 2 lines

you cant receive packets? Is there a concept of MAC address which you can pass to the hypervisor or can you run in promiscmous mode?

Right, without registering the IP address, you can not receive any

As the logical network interface has no own MAC address you actually

IP to the card. That also means, that without some additional effort,

like tcpdump fail and you need some patches in the dhcp tools. You can define options for routers to get more than your own packages,

IIRC you can only define a primary and secondary router per port.

Quoted 2 lines

Another question: When that driver runs for the physical card - it runs

The driver never ever runs for the physical card. It runs for 3 devices addresses, which are already behind a layer of virtualization and

a logical IP stack in the physical card. To Linux it looks like a

card.

Quoted 2 lines

in the context of a specific VM, correct? In other words it would be impossible to see the "card" of another instance?

By VM you mean virtual machine? Right. We are talking about real hardware

emulation. Think about it as VMWARE in hardware or with hardware support. So you have no access to logical cards of other Linuxes.
One difference to real hardware is, that multiple Linuxes have the same

address. The physical OSA network card and z/VM ensure, that incoming packets are delivered to the right Linux.

cheers
Christian

Frank, please correct me, if I am wrong....

Quoted 5 lines · jamal

Quoted 3 lines · Christian Bornträger

I am trying a small simplification here:
Each physical network adapter offers hundreds of device addresses. You need 3 of them to have one logical network adapter(read,write,data).

Quoted 3 lines

the "card" concept is what you call network adapter, correct? I take it that read and write are control channels and data is where the skb comes through?

don't ask me about naming....

Quoted 8 lines

Quoted 5 lines

S/390 has
hardware supported virtualization. Therefore can then use the hypervisor (LPAR or z/VM) to give specific LPARs or VM guests exactly 3 device addresses out of these hundreds.

Can you provision multiple of these cards per VM? if yes, is there some ID that will break it down to OSInstance:cardid?

Right, without registering the IP address, you can not receive any packet.

As the logical network interface has no own MAC address you actually speak IP to the card. That also means, that without some additional effort, tools like tcpdump fail and you need some patches in the dhcp tools.

You can define options for routers to get more than your own packages, but IIRC you can only define a primary and secondary router per port.

Quoted 2 lines

Quoted 2 lines

Can you provision multiple of these cards per VM? if yes, is there some ID that will break it down to OSInstance:cardid?

You did not answer this question.
Let me draw a diagram to show what i think the hierachy is:

Physical Card: MAC address X

+--- OSInstance A

| +-- "CARD" with IP A
| +-- "CARD" with IP B
| +-- "CARD" with IP C
| +-- "CARD" with IP D

.
.
.

+--- OSInstance N

+-- "CARD" with IP Z

Is the above reflective of what happens?

In other words, packet comes from the wire (with MAC address X); somehow the hypervisor(?) or firmware figures based on IP address A (assuming no other instance has that IP) it has to send packet to OSInstanceA.

OSInstanceA then selects further the CARD based on something probably in a descriptor?

Let me get to the point:
I think it would make sense for the "CARD" to be just another netdevice (call it "card" netdevice for this discussion).
The representation of the physical card in the OSInstance is also a netdevice(call it physical netdevice for this discussion) as it is now (excpet it has no IP address ever).
The "card netdevices" are stacked on top of the physical netdevice. This would be like an upside down bridge stacking relationship of netdevices....
It actually is no different from a few tunnel netdevices that sit on top of say eth0 or multiple PPP devices on top of ethx in a PPPOE relationship.
The demuxing for incoming packets is done at physical card netdevice to select the "card" netdevice whose receive method is then called. Reverse direction for transmit (we could go into details later, just wanna make sure this is sensible to begin with).

Does this sound reasonable? If yes, then if you do this you wont need to hack anything like IPV6 etc in your driver - they become merely netdevices. It should also allow for all standard features like ifconfig up/down etc of the "card" and setting IP addresses, VLANS etc to work as is. And you wont need to put any speacilized code in the driver. If its off tangent, then i just wasted 1/2 a cup of coffee energy typing away ;->

Quoted 3 lines

Right, without registering the IP address, you can not receive any packet.

If this is firmware issue, it would be wise to fix it. You should be able to register multiple MAC addresses hidden in the firmware (not at the Linux level) and have your "cards" netdevice use them. i.e the "card" netdevices would own those.

Quoted 4 lines

As the logical network interface has no own MAC address you actually speak IP to the card. That also means, that without some additional effort, tools like tcpdump fail and you need some patches in the dhcp tools.

Refer to above. If you actually have your virtual/"card" netdevice on top of the physical netdevice have a MAC address, then all tools should work as is with zero changes IMO.

Quoted 2 lines

You can define options for routers to get more than your own packages, but IIRC you can only define a primary and secondary router per port.

Ok, this is new information - what does a "router" mean in relation to a port?

Quoted 6 lines

Frank, please correct me, if I am wrong....

Quoted 3 lines · jamal

Quoted 2 lines · Christian Bornträger

I am trying a small simplification here:
Each physical network adapter offers hundreds of device addresses.

Quoted 1 line

Quoted 1 line

Quoted 1 line

need 3 of them to have one logical network

Quoted 3 lines

Quoted 3 lines

the "card" concept is what you call network adapter, correct? I take it that read and write are control channels and data is where

Quoted 3 lines

Quoted 1 line

skb comes through?

don't ask me about naming....

thats fine,
I think it doesnt matter what they are used for; important part is you need all 3 addresses to have a "card"; so got it.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

S/390 has
hardware supported virtualization. Therefore can then use the hypervisor (LPAR or z/VM) to give specific LPARs or VM guests

Quoted 3 lines

Quoted 3 lines

Quoted 1 line

device addresses out of these hundreds.

Can you provision multiple of these cards per VM? if yes, is there

Quoted 1 line

Quoted 1 line

ID that will break it down to OSInstance:cardid?

You did not answer this question.
Let me draw a diagram to show what i think the hierachy is:

Physical Card: MAC address X
|
|
+--- OSInstance A
| |
| +-- "CARD" with IP A
| +-- "CARD" with IP B
| +-- "CARD" with IP C
| +-- "CARD" with IP D
.
.
.
|
+--- OSInstance N
|
+-- "CARD" with IP Z

Is the above reflective of what happens?

In other words, packet comes from the wire (with MAC address X); somehow the hypervisor(?) or firmware figures based on IP address A (assuming no other instance has that IP) it has to send packet to OSInstanceA. OSInstanceA then selects further the CARD based on something probably in a descriptor?

Let me get to the point:
I think it would make sense for the "CARD" to be just another netdevice (call it "card" netdevice for this discussion).
The representation of the physical card in the OSInstance is also a netdevice(call it physical netdevice for this discussion) as it is now (excpet it has no IP address ever).
The "card netdevices" are stacked on top of the physical netdevice. This would be like an upside down bridge stacking relationship of netdevices....

From Linux frank1 point of view it has two pyhsical devices ,eth0 and eth1

It actually is no different from a few tunnel netdevices that sit on top of say eth0 or multiple PPP devices on top of ethx in a PPPOE relationship.

The demuxing for incoming packets is done at physical card netdevice to select the "card" netdevice whose receive method is then called. Reverse direction for transmit (we could go into details later, just wanna make sure this is sensible to begin with).
Does this sound reasonable? If yes, then if you do this you wont need to hack anything like IPV6 etc in your driver - they become merely netdevices. It should also allow for all standard features like ifconfig up/down etc of the "card" and setting IP addresses, VLANS etc to work as is. And you wont need to put any speacilized code in the driver. If its off tangent, then i just wasted 1/2 a cup of coffee energy typing away ;->

Quoted 2 lines

Right, without registering the IP address, you can not receive any

If this is firmware issue, it would be wise to fix it. You should be able to register multiple MAC addresses hidden in the firmware (not at the Linux level) and have your "cards" netdevice use them. i.e the "card" netdevices would own those.

Quoted 1 line

As the logical network interface has no own MAC address you actually

Quoted 1 line

IP to the card. That also means, that without some additional effort,

Quoted 1 line

like tcpdump fail and you need some patches in the dhcp tools.

Mit freundlichen Grüssen / Best regards
Frank Pavlic

Let's try again solvign the EUI64 generation for S/390. I looked over the IBM patch and I think it can be done a lot simpler:

- put a dev_id field in struct net_device, so that it uses space that would be wasted by padding otherwise.
- if this fields is non-null let ipv6_generate_eui64 use the algorithm from the QETH code to generate an EUI that's different for each OS instance. See code comments for details.

Quoted 8 lines · On Sun, 16 Jan 2005 12:54:31 +0100 Christoph Hellwig <[email protected]> wrote:

Let's try again solvign the EUI64 generation for S/390. I looked over the IBM patch and I think it can be done a lot simpler:

- put a dev_id field in struct net_device, so that it uses space that would be wasted by padding otherwise.
- if this fields is non-null let ipv6_generate_eui64 use the algorithm from the QETH code to generate an EUI that's different for each OS instance. See code comments for details.

I'll apply this patch.

It might be nice to genericize that huge comment. This will likely be not the last time we see devices which have this characteristic.

Refer to my comments:
any device that ends up having to do that is a hack in my opinion;-> The dev_id maybe useful for other things. the qeth driver needs fixing so it can stack the per-os instance interfaces.

At least Christian's patch is 100 times cleaner than what IBM has proposed. People have been "talking" endlessly about providing a clean version of the infrastructure necessary to support what qeth needs for EUI64 generation, but everyone did nothing but talk and suggest.