Skip to contentSkip to conversation

Hold reference to device_node during EEH event handling

PATCH In review Mike Mason 3 participants 7 messages Jul 15, 2009 – Jul 23, 2009

Subsystems (2)
lore
Share & tools
lore

Mike Mason

author
#0 lore raw

PATCH 1/1 Hold reference to device_node during EEH event handling

This patch increments the device_node reference counter when an EEH error occurs and decrements the counter when the event has been handled. This is to prevent the device_node from being released until eeh_event_handler() has had a chance to deal with the event. We've seen cases where the device_node is released too soon when an EEH event occurs during a dlpar remove, causing the event handler to attempt to access bad memory locations.

Please review and let me know of any concerns.

arch/powerpc/platforms/pseries/eeh_event.c +0 -0
@@ -75,6 +75,14 @@ static int eeh_event_handler(void * dumm
	if (event == NULL)
		return 0;

+ /* EEH holds a reference to the device_node, so if it
+ * equals 1 it's no longer valid and the event should
+ * be ignored */
+ if (atomic_read(&event->dn->kref.refcount) == 1) {
+ of_node_put(event->dn);
+ return 0;
+ }
+

	/* Serialize processing of EEH events */
	mutex_lock(&eeh_event_mutex);
	eeh_mark_slot(event->dn, EEH_MODE_RECOVERING);

@@ -86,6 +94,7 @@ static int eeh_event_handler(void * dumm

	eeh_clear_slot(event->dn, EEH_MODE_RECOVERING);
	pci_dev_put(event->dev);
+	of_node_put(event->dn);
	kfree(event);
	mutex_unlock(&eeh_event_mutex);

@@ -140,7 +149,7 @@ int eeh_send_failure_event (struct devic

	if (dev)
		pci_dev_get(dev);

- event->dn = dn;

+	event->dn = of_node_get(dn);
	event->dev = dev;

	/* We may or may not be called in an interrupt context */
#1 lore raw
Quoted 9 lines · Mike Mason

This patch increments the device_node reference counter when an EEH error occurs and decrements the counter when the event has been handled. This is to prevent the device_node from being released until eeh_event_handler() has had a chance to deal with the event. We've seen cases where the device_node is released too soon when an EEH event occurs during a dlpar remove, causing the event handler to attempt to access bad memory locations.

Please review and let me know of any concerns.

Taking a reference sounds sane, but ...

Quoted 5 lines

arch/powerpc/platforms/pseries/eeh_event.c static int eeh_event_handler(void * dumm · line 84

Quoted 10 lines
arch/powerpc/platforms/pseries/eeh_event.c +7 -0
@@ -75,6 +75,14 @@ static int eeh_event_handler(void * dumm
 	if (event == NULL)
 		return 0;
 
+	/* EEH holds a reference to the device_node, so if it
+	 * equals 1 it's no longer valid and the event should
+	 * be ignored */
+	if (atomic_read(&event->dn->kref.refcount) == 1) {
+		of_node_put(event->dn);
+		return 0;
+	}

That's really gross :)

And what happens if the refcount goes to 1 just after the check? ie. here.

arch/powerpc/platforms/pseries/eeh_event.c static int eeh_event_handler(void * dumm · line 87

Quoted 3 lines
arch/powerpc/platforms/pseries/eeh_event.c +0 -0
 	/* Serialize processing of EEH events */
 	mutex_lock(&eeh_event_mutex);
 	eeh_mark_slot(event->dn, EEH_MODE_RECOVERING);

cheers

author
#2 lore raw
Quoted 21 lines · Michael Ellerman
Quoted 9 lines · Mike Mason

This patch increments the device_node reference counter when an EEH error occurs and decrements the counter when the event has been handled. This is to prevent the device_node from being released until eeh_event_handler() has had a chance to deal with the event. We've seen cases where the device_node is released too soon when an EEH event occurs during a dlpar remove, causing the event handler to attempt to access bad memory locations.

Please review and let me know of any concerns.

Taking a reference sounds sane, but ...

Quoted 8 lines
arch/powerpc/platforms/pseries/eeh_event.c +0 -0
@@ -75,6 +75,14 @@ static int eeh_event_handler(void * dumm
 	if (event == NULL)
 		return 0;
 
Quoted 9 lines

arch/powerpc/platforms/pseries/eeh_event.c static int eeh_event_handler(void * dumm · line 84

Quoted 7 lines
arch/powerpc/platforms/pseries/eeh_event.c +7 -0
+	/* EEH holds a reference to the device_node, so if it
+	 * equals 1 it's no longer valid and the event should
+	 * be ignored */
+	if (atomic_read(&event->dn->kref.refcount) == 1) {
+		of_node_put(event->dn);
+		return 0;
+	}

That's really gross :)

Agreed. I'll look for another way to determine if device is gone and the event should be ignored. Suggestions are welcome :-)

Quoted 11 lines

And what happens if the refcount goes to 1 just after the check? ie. here.

arch/powerpc/platforms/pseries/eeh_event.c static int eeh_event_handler(void * dumm · line 87

Quoted 3 lines
arch/powerpc/platforms/pseries/eeh_event.c +0 -0
 	/* Serialize processing of EEH events */
 	mutex_lock(&eeh_event_mutex);
 	eeh_mark_slot(event->dn, EEH_MODE_RECOVERING);

cheers

#3 lore raw
Quoted 32 lines · Mike Mason
Quoted 21 lines · Michael Ellerman
Quoted 9 lines · Mike Mason

This patch increments the device_node reference counter when an EEH error occurs and decrements the counter when the event has been handled. This is to prevent the device_node from being released until eeh_event_handler() has had a chance to deal with the event. We've seen cases where the device_node is released too soon when an EEH event occurs during a dlpar remove, causing the event handler to attempt to access bad memory locations.

Please review and let me know of any concerns.

Taking a reference sounds sane, but ...

Quoted 8 lines
arch/powerpc/platforms/pseries/eeh_event.c +0 -0
@@ -75,6 +75,14 @@ static int eeh_event_handler(void * dumm
 	if (event == NULL)
 		return 0;
 
Quoted 9 lines

arch/powerpc/platforms/pseries/eeh_event.c static int eeh_event_handler(void * dumm · line 84

Quoted 7 lines
arch/powerpc/platforms/pseries/eeh_event.c +7 -0
+	/* EEH holds a reference to the device_node, so if it
+	 * equals 1 it's no longer valid and the event should
+	 * be ignored */
+	if (atomic_read(&event->dn->kref.refcount) == 1) {
+		of_node_put(event->dn);
+		return 0;
+	}

That's really gross :)

Quoted 2 lines

Agreed. I'll look for another way to determine if device is gone and the event should be ignored. Suggestions are welcome :-)

Benh and I had a quick chat about it, and were wondering whether what you really should be doing is taking a reference to the pci device (perhaps as well as the device node).

@@ -140,7 +149,7 @@ int eeh_send_failure_event (struct devic

        if (dev)
                pci_dev_get(dev);

- event->dn = dn;

+       event->dn = of_node_get(dn);
        event->dev = dev;

pci devs are refcounted too, see pci_dev_get(), so taking a reference there would be the "right" thing to do - otherwise there's no guarantee it still exists later, unless there's some other trick in the EEH code.

Taking a reference would presumably block a concurrent hotunplug until you'd processed the EEH event and dropped your reference. That might be OK, or you could add a hotplug notifier to the EEH code and drop the reference there and mark the event as handled or something.

All of that with the caveat that I don't really know the EEH or hotplug code :D

cheers

author
#4 lore raw
Quoted 30 lines · Michael Ellerman
Quoted 29 lines · Mike Mason
Quoted 28 lines · Michael Ellerman
Quoted 9 lines · Mike Mason

This patch increments the device_node reference counter when an EEH error occurs and decrements the counter when the event has been handled. This is to prevent the device_node from being released until eeh_event_handler() has had a chance to deal with the event. We've seen cases where the device_node is released too soon when an EEH event occurs during a dlpar remove, causing the event handler to attempt to access bad memory locations.

Please review and let me know of any concerns.

Taking a reference sounds sane, but ...

Quoted 5 lines

arch/powerpc/platforms/pseries/eeh_event.c static int eeh_event_handler(void * dumm · line 84

Quoted 10 lines
arch/powerpc/platforms/pseries/eeh_event.c +7 -0
@@ -75,6 +75,14 @@ static int eeh_event_handler(void * dumm
 	if (event == NULL)
 		return 0;
 
+	/* EEH holds a reference to the device_node, so if it
+	 * equals 1 it's no longer valid and the event should
+	 * be ignored */
+	if (atomic_read(&event->dn->kref.refcount) == 1) {
+		of_node_put(event->dn);
+		return 0;
+	}

That's really gross :)

Quoted 2 lines
Quoted 2 lines

Agreed. I'll look for another way to determine if device is gone and the event should be ignored. Suggestions are welcome :-)

Actually, it turns out the atomic_read() isn't necessary. I just need to take the reference to the device_node when the EEH error is detected and let EEH try to handle the error. EEH detects the fact that the device is no longer valid, aborts the recovery attempt, then gives the device_node reference back. Works as expected.

I'll resubmit the patch without the atomic_read().

Quoted 4 lines

Benh and I had a quick chat about it, and were wondering whether what you really should be doing is taking a reference to the pci device (perhaps as well as the device node).

EEH already does that 3 lines before the of_node_get (see below).

Review int eeh_send_failure_event (struct devic · line 154

Quoted 9 lines
(quoted diff) +1 -1
@@ -140,7 +149,7 @@ int eeh_send_failure_event (struct devic
        if (dev)
                pci_dev_get(dev);
 
-       event->dn = dn;
+       event->dn = of_node_get(dn);
        event->dev = dev;
 

Thanks,
Mike

#5 lore raw
Quoted 31 lines · Mike Mason
Quoted 30 lines · Michael Ellerman
Quoted 29 lines · Mike Mason
Quoted 28 lines · Michael Ellerman
Quoted 9 lines · Mike Mason

This patch increments the device_node reference counter when an EEH error occurs and decrements the counter when the event has been handled. This is to prevent the device_node from being released until eeh_event_handler() has had a chance to deal with the event. We've seen cases where the device_node is released too soon when an EEH event occurs during a dlpar remove, causing the event handler to attempt to access bad memory locations.

Please review and let me know of any concerns.

Taking a reference sounds sane, but ...

Quoted 5 lines

arch/powerpc/platforms/pseries/eeh_event.c static int eeh_event_handler(void * dumm · line 84

Quoted 10 lines
arch/powerpc/platforms/pseries/eeh_event.c +7 -0
@@ -75,6 +75,14 @@ static int eeh_event_handler(void * dumm
 	if (event == NULL)
 		return 0;
 
+	/* EEH holds a reference to the device_node, so if it
+	 * equals 1 it's no longer valid and the event should
+	 * be ignored */
+	if (atomic_read(&event->dn->kref.refcount) == 1) {
+		of_node_put(event->dn);
+		return 0;
+	}

That's really gross :)

Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

Agreed. I'll look for another way to determine if device is gone and the event should be ignored. Suggestions are welcome :-)

Actually, it turns out the atomic_read() isn't necessary. I just need to take the reference to the device_node when the EEH error is detected and let EEH try to handle the error. EEH detects the fact that the device is no longer valid, aborts the recovery attempt, then gives the device_node reference back. Works as expected.

How does it detect that the device is no longer valid?

Quoted 8 lines

I'll resubmit the patch without the atomic_read().

Quoted 4 lines

Benh and I had a quick chat about it, and were wondering whether what you really should be doing is taking a reference to the pci device (perhaps as well as the device node).

EEH already does that 3 lines before the of_node_get (see below).

Ah right, while you're touching the code, mind changing it to the simpler and more obvious:

Quoted 1 line
        event->dev = pci_dev_get(dev);

cheers

#6 lore raw
In reply to Michael Ellerman

2009/7/16 Michael Ellerman <[email protected]>:

Quoted 6 lines · Mike Mason
Quoted 5 lines
Quoted 4 lines · Michael Ellerman
Quoted 3 lines · Mike Mason

This patch increments the device_node reference counter when an EEH error occurs and decrements the counter when the event has been handled. =C2=A0This is to prevent the device_node from being released=

until

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

eeh_event_handler() has had a chance to deal with the event. =C2=A0We=

've

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines

seen cases where the device_node is released too soon when an EEH event occurs during a dlpar remove, causing the event handler to attempt to access bad memory locations.

Please review and let me know of any concerns.

Taking a reference sounds sane, but ...

Quoted 3 lines

--- a/arch/powerpc/platforms/pseries/eeh_event.c =C2=A0 2008-10-09 15=

:13:53.000000000 -0700

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

+++ b/arch/powerpc/platforms/pseries/eeh_event.c =C2=A0 2009-07-14 14=

:14:00.000000000 -0700

Quoted 14 lines
Quoted 14 lines
Quoted 13 lines

Review static int eeh_event_handler(void * dumm · line 84

Quoted 11 lines
(quoted diff) +7 -0
@@ -75,6 +75,14 @@ static int eeh_event_handler(void * dumm
 =C2=A0 =C2=A0if (event =3D=3D NULL)
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return 0;
 
+ =C2=A0/* EEH holds a reference to the device_node, so if it
+ =C2=A0 * equals 1 it's no longer valid and the event should
+ =C2=A0 * be ignored */
+ =C2=A0if (atomic_read(&event->dn->kref.refcount) =3D=3D 1) {
+ =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0of_node_put(event->dn);
+ =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return 0;
+ =C2=A0}

That's really gross :)

Quoted 1 line
Quoted 1 line

Agreed. =C2=A0I'll look for another way to determine if device is gone a=

nd

Quoted 14 lines
Quoted 1 line

the event should be ignored. =C2=A0Suggestions are welcome :-)

Benh and I had a quick chat about it, and were wondering whether what you really should be doing is taking a reference to the pci device (perhaps as well as the device node).

(quoted diff) +1 -1
@@ -140,7 +149,7 @@ int eeh_send_failure_event (struct devic
 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (dev)
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0pci_dev_get(dev);
 
- =C2=A0 =C2=A0 =C2=A0 event->dn =3D dn;
+ =C2=A0 =C2=A0 =C2=A0 event->dn =3D of_node_get(dn);
 =C2=A0 =C2=A0 =C2=A0 =C2=A0event->dev =3D dev;
 
Quoted 3 lines

pci devs are refcounted too, see pci_dev_get(), so taking a reference there would be the "right" thing to do - otherwise there's no guarantee it still exists later, unless there's some other trick in the EEH code.

I thought that the eeh code did pci gets and puts in the right locations, perhaps I (incorrectly) assumed that this meant that the of_dn use count never dropped to zero ...

I think my logic was:
-- pci device init does of_node_get
-- pci device shutdown does of_node_put
-- pci device shutdown can never run as long as pci use count is > 0

Thus, explicit of_node_get was usually not needed.

So, for example, see above: I was figuring that the pci_dev_get(dev); was enough to protect the dn too .. although maybe if dev is null, then things go wrong ...

--linas