Skip to contentSkip to conversation

powerpc: ibmveth: Harden driver initilisation for kexec

PATCH In review Santiago Leon 5 participants 12 messages Mar 2, 2006 – Aug 29, 2006 2 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): 20060131042903.GF28896@krispykreme

#2 lore raw

PATCH 1/1 powerpc: ibmveth: Harden driver initilisation for kexec

From: Michael Ellerman <[email protected]>

After a kexec the veth driver will fail when trying to register with the Hypervisor because the previous kernel has not unregistered.

So if the registration fails, we unregister and then try again.

---

  drivers/net/ibmveth.c |   32 ++++++++++++++++++++++++++------
  1 files changed, 26 insertions(+), 6 deletions(-)

Looks good to me, and has been around for a couple of months.

drivers/net/ibmveth.c +25 -0
Index: kexec/drivers/net/ibmveth.c
===================================================================
@@ -436,6 +436,31 @@ static void ibmveth_cleanup(struct ibmve
  		ibmveth_free_buffer_pool(adapter, &adapter->rx_buff_pool[i]);
  }
 
+static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
+		union ibmveth_buf_desc rxq_desc, u64 mac_address)
+{
+	int rc, try_again = 1;
+
+	/* After a kexec the adapter will still be open, so our attempt to
+	 * open it will fail. So if we get a failure we free the adapter and
+	 * try again, but only once. */
+retry:
+	rc = h_register_logical_lan(adapter->vdev->unit_address,
+			adapter->buffer_list_dma, rxq_desc.desc,
+			adapter->filter_list_dma, mac_address);
+
+	if (rc != H_Success && try_again) {
+		do {
+			rc = h_free_logical_lan(adapter->vdev->unit_address);
+		} while (H_isLongBusy(rc) || (rc == H_Busy));
+
+		try_again = 0;
+		goto retry;
+	}
+
+	return rc;
+}
+
  static int ibmveth_open(struct net_device *netdev)
  {
  	struct ibmveth_adapter *adapter = netdev->priv;
@@ -504,12 +529,7 @@ static int ibmveth_open(struct net_devic
  	ibmveth_debug_printk("filter list @ 0x%p\n", adapter->filter_list_addr);
  	ibmveth_debug_printk("receive q   @ 0x%p\n", 

adapter->rx_queue.queue_addr);

-
- lpar_rc = h_register_logical_lan(adapter->vdev->unit_address,
- adapter->buffer_list_dma,
- rxq_desc.desc,
- adapter->filter_list_dma,
- mac_address);
+ lpar_rc = ibmveth_register_logical_lan(adapter, rxq_desc, mac_address);

if(lpar_rc != H_Success) {
ibmveth_error_printk("h_register_logical_lan failed with %ld\n",
lpar_rc);

#3 lore raw

Hi Jeff,

I realise it's late, but it'd be really good if you could send this up for 2.6.16, we're hosed without it.

cheers

Quoted 76 lines · Santiago Leon

From: Michael Ellerman <[email protected]>

After a kexec the veth driver will fail when trying to register with the Hypervisor because the previous kernel has not unregistered.

So if the registration fails, we unregister and then try again.

---

  drivers/net/ibmveth.c |   32 ++++++++++++++++++++++++++------
  1 files changed, 26 insertions(+), 6 deletions(-)

Looks good to me, and has been around for a couple of months.

Index: kexec/drivers/net/ibmveth.c
===================================================================

drivers/net/ibmveth.c +26 -6
@@ -436,6 +436,31 @@ static void ibmveth_cleanup(struct ibmve
  		ibmveth_free_buffer_pool(adapter, &adapter->rx_buff_pool[i]);
  }
 
+static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
+		union ibmveth_buf_desc rxq_desc, u64 mac_address)
+{
+	int rc, try_again = 1;
+
+	/* After a kexec the adapter will still be open, so our attempt to
+	 * open it will fail. So if we get a failure we free the adapter and
+	 * try again, but only once. */
+retry:
+	rc = h_register_logical_lan(adapter->vdev->unit_address,
+			adapter->buffer_list_dma, rxq_desc.desc,
+			adapter->filter_list_dma, mac_address);
+
+	if (rc != H_Success && try_again) {
+		do {
+			rc = h_free_logical_lan(adapter->vdev->unit_address);
+		} while (H_isLongBusy(rc) || (rc == H_Busy));
+
+		try_again = 0;
+		goto retry;
+	}
+
+	return rc;
+}
+
  static int ibmveth_open(struct net_device *netdev)
  {
  	struct ibmveth_adapter *adapter = netdev->priv;
@@ -504,12 +529,7 @@ static int ibmveth_open(struct net_devic
  	ibmveth_debug_printk("filter list @ 0x%p\n", adapter->filter_list_addr);
  	ibmveth_debug_printk("receive q   @ 0x%p\n",
 adapter->rx_queue.queue_addr);
 
-
-	lpar_rc = h_register_logical_lan(adapter->vdev->unit_address,
-					 adapter->buffer_list_dma,
-					 rxq_desc.desc,
-					 adapter->filter_list_dma,
-					 mac_address);
+	lpar_rc = ibmveth_register_logical_lan(adapter, rxq_desc, mac_address);
 
  	if(lpar_rc != H_Success) {

ibmveth_error_printk("h_register_logical_lan failed with %ld\n",
lpar_rc);

_______________________________________________
Linuxppc64-dev mailing list
[email protected]
https://ozlabs.org/mailman/listinfo/linuxppc64-dev

Signature · 8 lines
Michael Ellerman
IBM OzLabs

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person

_______________________________________________
Linuxppc64-dev mailing list
[email protected]
https://ozlabs.org/mailman/listinfo/linuxppc64-dev

#4 lore raw
Quoted 4 lines · On Fri, 3 Mar 2006 11:22:45 +1100 Michael Ellerman wrote:

Hi Jeff,

I realise it's late, but it'd be really good if you could send this up for 2.6.16, we're hosed without it.

I'm wondering if this means that for every virtual/hypervisor situation, we have to modify any $interested_drivers. Why wouldn't we come up with a cleaner solution (in the long term)?

E.g., could the hypervisor know when one of it's virtual OSes dies or reboots and release its resources then?

This patch just looks like a short-term solution to me.

Quoted 75 lines

cheers

Quoted 72 lines · Santiago Leon

From: Michael Ellerman <[email protected]>

After a kexec the veth driver will fail when trying to register with the Hypervisor because the previous kernel has not unregistered.

So if the registration fails, we unregister and then try again.

---

  drivers/net/ibmveth.c |   32 ++++++++++++++++++++++++++------
  1 files changed, 26 insertions(+), 6 deletions(-)

Looks good to me, and has been around for a couple of months.

Index: kexec/drivers/net/ibmveth.c
===================================================================

drivers/net/ibmveth.c +26 -6
@@ -436,6 +436,31 @@ static void ibmveth_cleanup(struct ibmve
  		ibmveth_free_buffer_pool(adapter, &adapter->rx_buff_pool[i]);
  }
 
+static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
+		union ibmveth_buf_desc rxq_desc, u64 mac_address)
+{
+	int rc, try_again = 1;
+
+	/* After a kexec the adapter will still be open, so our attempt to
+	 * open it will fail. So if we get a failure we free the adapter and
+	 * try again, but only once. */
+retry:
+	rc = h_register_logical_lan(adapter->vdev->unit_address,
+			adapter->buffer_list_dma, rxq_desc.desc,
+			adapter->filter_list_dma, mac_address);
+
+	if (rc != H_Success && try_again) {
+		do {
+			rc = h_free_logical_lan(adapter->vdev->unit_address);
+		} while (H_isLongBusy(rc) || (rc == H_Busy));
+
+		try_again = 0;
+		goto retry;
+	}
+
+	return rc;
+}
+
  static int ibmveth_open(struct net_device *netdev)
  {
  	struct ibmveth_adapter *adapter = netdev->priv;
@@ -504,12 +529,7 @@ static int ibmveth_open(struct net_devic
  	ibmveth_debug_printk("filter list @ 0x%p\n", adapter->filter_list_addr);
  	ibmveth_debug_printk("receive q   @ 0x%p\n",
 adapter->rx_queue.queue_addr);
 
-
-	lpar_rc = h_register_logical_lan(adapter->vdev->unit_address,
-					 adapter->buffer_list_dma,
-					 rxq_desc.desc,
-					 adapter->filter_list_dma,
-					 mac_address);
+	lpar_rc = ibmveth_register_logical_lan(adapter, rxq_desc, mac_address);
 
  	if(lpar_rc != H_Success) {

ibmveth_error_printk("h_register_logical_lan failed with %ld\n",
lpar_rc);

Quoted 8 lines
Quoted 4 lines

_______________________________________________
Linuxppc64-dev mailing list
[email protected]
https://ozlabs.org/mailman/listinfo/linuxppc64-dev

Signature · 2 lines
Michael Ellerman
IBM OzLabs

---
~Randy

#5 lore raw
Quoted 2 lines · Randy.Dunlap

E.g., could the hypervisor know when one of it's virtual OSes dies or reboots and release its resources then?

I think the point is that with kexec, the same virtual machine keeps running, so the hypervisor doesn't see the OS dying or rebooting.

Paul.

#6 lore raw
In reply to Randy.Dunlap
Quoted 6 lines · Randy.Dunlap
Quoted 4 lines · On Fri, 3 Mar 2006 11:22:45 +1100 Michael Ellerman wrote:

Hi Jeff,

I realise it's late, but it'd be really good if you could send this up for 2.6.16, we're hosed without it.

Quoted 6 lines

I'm wondering if this means that for every virtual/hypervisor situation, we have to modify any $interested_drivers. Why wouldn't we come up with a cleaner solution (in the long term)?

E.g., could the hypervisor know when one of it's virtual OSes dies or reboots and release its resources then?

It does exactly that for a regular reboot, but when we kexec we _don't_ die or reboot, as far as the Hypervisor is concerned it's all systems go.

It's something of a double-edged sword, we're totally in control which gives us lots of flexibility, and _fast_ reboot times, but we also have to do a bit of extra stuff (ie. this patch) to keep things sane.

cheers

Signature · 8 lines
Michael Ellerman
IBM OzLabs

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person

_______________________________________________
Linuxppc64-dev mailing list
[email protected]
https://ozlabs.org/mailman/listinfo/linuxppc64-dev

#7 lore raw
Quoted 14 lines · On Fri, 3 Mar 2006 12:10:47 +1100 Michael Ellerman wrote:
Quoted 6 lines · Randy.Dunlap
Quoted 4 lines · On Fri, 3 Mar 2006 11:22:45 +1100 Michael Ellerman wrote:

Hi Jeff,

I realise it's late, but it'd be really good if you could send this up for 2.6.16, we're hosed without it.

Quoted 6 lines

I'm wondering if this means that for every virtual/hypervisor situation, we have to modify any $interested_drivers. Why wouldn't we come up with a cleaner solution (in the long term)?

E.g., could the hypervisor know when one of it's virtual OSes dies or reboots and release its resources then?

Quoted 6 lines

It does exactly that for a regular reboot, but when we kexec we _don't_ die or reboot, as far as the Hypervisor is concerned it's all systems go.

It's something of a double-edged sword, we're totally in control which gives us lots of flexibility, and _fast_ reboot times, but we also have to do a bit of extra stuff (ie. this patch) to keep things sane.

s/this patch/some patch/

Yes, you have certainly thought about this more/longer than I have, so why is something more generic like this bad instead of good:

Somewhere early in start_kernel() (e.g.), do an hv call that says "free all assigned resources".

Maybe hv doesn't know "all assigned resources."
Maybe it's just that this patch is simpler than an hv change, although this (current) patch could leave some other drivers that need to be "fixed," while an hv change wouldn't do that.

So I'm not opposed to this current patch as a short-term solution, but I don't think it's the right long-term solution.

---
~Randy

#8 lore raw
In reply to Santiago Leon
Quoted 10 lines · Santiago Leon

From: Michael Ellerman <[email protected]>

After a kexec the veth driver will fail when trying to register with the Hypervisor because the previous kernel has not unregistered.

So if the registration fails, we unregister and then try again.

Quoted 6 lines

---

 drivers/net/ibmveth.c |   32 ++++++++++++++++++++++++++------
 1 files changed, 26 insertions(+), 6 deletions(-)

Looks good to me, and has been around for a couple of months.

This seems completely bonkers to me:

are resources available?
if no
free resources
try again

It makes resource checking pointless.

Jeff

#9 lore raw
Quoted 16 lines · Jeff Garzik
Quoted 15 lines · Santiago Leon

From: Michael Ellerman <[email protected]>

After a kexec the veth driver will fail when trying to register with the Hypervisor because the previous kernel has not unregistered.

So if the registration fails, we unregister and then try again.

---

 drivers/net/ibmveth.c |   32 ++++++++++++++++++++++++++------
 1 files changed, 26 insertions(+), 6 deletions(-)
Quoted 8 lines
Quoted 1 line

Looks good to me, and has been around for a couple of months.

This seems completely bonkers to me:

are resources available?
if no
free resources
try again

I'm not sure I follow, are you suggesting we do the h_free_logical_lan() unconditionally, followed by h_register_logical_lan() ??

If that's what you mean, I didn't do it that way because it would effect the normal code path. This patch only modifies the behaviour if we fail to register the adapter. I'm much more comfortable changing the failure case than the default.

cheers

Signature · 8 lines
Michael Ellerman
IBM OzLabs

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person

_______________________________________________
Linuxppc64-dev mailing list
[email protected]
https://ozlabs.org/mailman/listinfo/linuxppc64-dev

#10 lore raw
In reply to Santiago Leon
Quoted 7 lines · Santiago Leon

From: Michael Ellerman <[email protected]>

After a kexec the veth driver will fail when trying to register with the Hypervisor because the previous kernel has not unregistered.

So if the registration fails, we unregister and then try again.

Quoted 7 lines

---

  drivers/net/ibmveth.c |   32 ++++++++++++++++++++++++++------
  1 files changed, 26 insertions(+), 6 deletions(-)

Looks like this hit the floor. Any chance of getting it into to 2.6.17 Jeff? AFAICT it should still apply cleanly.

cheers

Signature · 8 lines
Michael Ellerman
IBM OzLabs

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person
#11 lore raw
Quoted 13 lines · Michael Ellerman
Quoted 12 lines · Santiago Leon

From: Michael Ellerman <[email protected]>

After a kexec the veth driver will fail when trying to register with the Hypervisor because the previous kernel has not unregistered.

So if the registration fails, we unregister and then try again.

---

Quoted 5 lines
Quoted 2 lines
  drivers/net/ibmveth.c |   32 ++++++++++++++++++++++++++------
  1 files changed, 26 insertions(+), 6 deletions(-)

Looks like this hit the floor. Any chance of getting it into to 2.6.17 Jeff? AFAICT it should still apply cleanly.

/me knocks politely

Signature · 8 lines
Michael Ellerman
IBM OzLabs

wwweb: http://michael.ellerman.id.au
phone: +61 2 6212 1183 (tie line 70 21183)

We do not inherit the earth from our ancestors,
we borrow it from our children. - S.M.A.R.T Person
author Acked-by Anton Blanchard
#12 lore raw
Quoted 4 lines · Michael Ellerman
Quoted 2 lines

Looks like this hit the floor. Any chance of getting it into to 2.6.17 Jeff? AFAICT it should still apply cleanly.

/me knocks politely

Actually, it doesn't apply cleanly anymore. Here's a patch that does.

Signature · 3 lines
Santiago A. Leon
Power Linux Development
IBM Linux Technology Center

From: Michael Ellerman <[email protected]>

After a kexec the ibmveth driver will fail when trying to register with the Hypervisor because the previous kernel has not unregistered.

So if the registration fails, we unregister and then try again.

 ibmveth.c |   31 ++++++++++++++++++++++++++-----
 1 file changed, 26 insertions(+), 5 deletions(-)
drivers/net/ibmveth.c +26 -5
Index: kexec/drivers/net/ibmveth.c
===================================================================
@@ -436,6 +436,31 @@ static void ibmveth_cleanup(struct ibmve
 		ibmveth_free_buffer_pool(adapter, &adapter->rx_buff_pool[i]);
 }
 
+static int ibmveth_register_logical_lan(struct ibmveth_adapter *adapter,
+        union ibmveth_buf_desc rxq_desc, u64 mac_address)
+{
+	int rc, try_again = 1;
+
+	/* After a kexec the adapter will still be open, so our attempt to
+	* open it will fail. So if we get a failure we free the adapter and
+	* try again, but only once. */
+retry:
+	rc = h_register_logical_lan(adapter->vdev->unit_address,
+				    adapter->buffer_list_dma, rxq_desc.desc,
+				    adapter->filter_list_dma, mac_address);
+
+	if (rc != H_SUCCESS && try_again) {
+		do {
+			rc = h_free_logical_lan(adapter->vdev->unit_address);
+		} while (H_IS_LONG_BUSY(rc) || (rc == H_BUSY));
+
+		try_again = 0;
+		goto retry;
+	}
+
+	return rc;
+}
+
 static int ibmveth_open(struct net_device *netdev)
 {
 	struct ibmveth_adapter *adapter = netdev->priv;
@@ -505,11 +530,7 @@ static int ibmveth_open(struct net_devic
 	ibmveth_debug_printk("receive q   @ 0x%p\n", adapter->rx_queue.queue_addr);
 
     
-	lpar_rc = h_register_logical_lan(adapter->vdev->unit_address,
-					 adapter->buffer_list_dma,
-					 rxq_desc.desc,
-					 adapter->filter_list_dma,
-					 mac_address);
+	lpar_rc = ibmveth_register_logical_lan(adapter, rxq_desc, mac_address);
 
 	if(lpar_rc != H_SUCCESS) {
 		ibmveth_error_printk("h_register_logical_lan failed with %ld\n", lpar_rc);
#13 lore raw
Quoted 2 lines · Santiago Leon
Quoted 1 line · Michael Ellerman

Any chance of getting it into to 2.6.17 ...

2.6.19 perhaps?

cheers