OpenSM 3.3.16 at 100% CPU load, "console off"

From strace:

Hi Hal,

we've encountered an issue with OpenSM 3.3.16 and the config option "console off".
OpenSM processes are at 100% CPU load.

Quoted 1 line

From strace:

poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
read(0, "", 4096)                       = 0
poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
read(0, "", 4096)                       = 0
poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
read(0, "", 4096)                       = 0

As far as I've seen in the code, the function osm_console() from opensm/osm_console.c is the only function which uses poll().

Is this issue already known or perhaps already fixed?

Thanks,
Sebastian

Quoted 7 lines

Quoted 1 line

From strace:

poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
read(0, "", 4096)                       = 0
poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
read(0, "", 4096)                       = 0
poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
read(0, "", 4096)                       = 0

So this doesn't block for 1 second and that's why the CPU is 100% ?

Quoted 8 lines · Hal Rosenstock

Quoted 7 lines

Quoted 1 line

From strace:

poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
read(0, "", 4096)                       = 0
poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
read(0, "", 4096)                       = 0
poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
read(0, "", 4096)                       = 0

Quoted 1 line

So this doesn't block for 1 second and that's why the CPU is 100% ?

Looks like it is spinning on a closed socket (or stdin) -- calling poll() on such will return immediately...

Hi Sebastian,

Quoted 6 lines · Sebastian Riemer

Hi Hal,

we've encountered an issue with OpenSM 3.3.16 and the config option "console off".
OpenSM processes are at 100% CPU load.

Quoted 7 lines

Quoted 1 line

From strace:

poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
read(0, "", 4096)                       = 0
poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
read(0, "", 4096)                       = 0
poll([{fd=0, events=POLLIN}], 1, 1000)  = 1 ([{fd=0, revents=POLLIN}])
read(0, "", 4096)                       = 0

So this doesn't block for 1 second and that's why the CPU is 100% ?

Quoted 2 lines

As far as I've seen in the code, the function osm_console() from opensm/osm_console.c is the only function which uses poll().

osm_vendor_ibumad has a receiver thread polling umad under the covers of umad_recv but I think that uses infinite rather than 1 second.

Quoted 1 line

Is this issue already known or perhaps already fixed?

This area of the console code has not changed in quite a while. Any idea if this works with older versions of OpenSM ?

This is first I've heard of this issue.

-- Hal

Quoted 3 lines

Thanks,
Sebastian

Do you recall the sequence to get to this ?

Was console option changed to off and then OpenSM SIGHUP'd ? Something else ?

Is this reproducible ?

Quoted 6 lines · Hal Rosenstock

Do you recall the sequence to get to this ?

Was console option changed to off and then OpenSM SIGHUP'd ? Something else ?

Is this reproducible ?

Yes, now I can reproduce it. The opensm has been initially started with "console off" and I activate "console local" and restart the service. CPU load is at 100% immediately. I set "console off" again and restart the service and CPU load is low again.

I did this three times in a row, now. And the third time it even remained at 100% load in the "off" state. I've set "local" and "off" again and CPU load was low again.

Cheers,
Sebastian

What does service restart do in terms of OpenSM ?

Note that the console parameter is _not_ changeable "on the fly" right now so if OpenSM is being SIGHUP'd by service restart then this is a current limitation (and is clearly not detected/protected against in the current code base). It sounds like that may be what is going on.

Quoted 6 lines · Hal Rosenstock

What does service restart do in terms of OpenSM ?

Note that the console parameter is _not_ changeable "on the fly" right now so if OpenSM is being SIGHUP'd by service restart then this is a current limitation (and is clearly not detected/protected against in the current code base). It sounds like that may be what is going on.

Yes, it emits SIGHUP. Thanks for the information! The opensm is a critical component. So IMHO it needs to be fixed in a way that it either protects itself against such changes by ignoring them on the fly or it needs to support these changes.

The current situation is not really acceptable and the opensm stability is crucial. So I'll think about fixing it.
Are you interested in patches in this regard?

Cheers,
Sebastian