Skip to contentSkip to conversation

openbmc Digest, Vol 60, Issue 27

DISCUSSION Manikandan 1 participant 1 message Aug 8, 2020 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Manikandan

author
#1 lore raw
Quoted 117 lines · On Sat, Aug 08, 2020 at 12:00:04PM +1000, [email protected]…

Send openbmc mailing list submissions to
[email protected]

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.ozlabs.org/listinfo/openbmc
or, via email, send a message with subject or body 'help' to
[email protected]

You can reach the person managing the list at
[email protected]

When replying, please edit your Subject line so it is more specific than "Re: Contents of openbmc digest..."

Today's Topics:

1. Re: Inconsistent performance of dbus call GetManagedObjects to PSUSensor in dbus-sensors (Ed Tanous)
2. RE: system power control (Zhao Kun)

----------------------------------------------------------------------

Message: 1
Date: Fri, 7 Aug 2020 16:17:20 -0700
From: Ed Tanous <[email protected]>
To: Alex Qiu <[email protected]>
Cc: OpenBMC Maillist <[email protected]>, James Feist
<[email protected]>, Peter Lundgren <[email protected]>, Josh Lehan <[email protected]>, Jason Ling <[email protected]>, Sui Chen <[email protected]>, Jie Yang <[email protected]>, Drew Macrae <[email protected]>
Subject: Re: Inconsistent performance of dbus call GetManagedObjects
to PSUSensor in dbus-sensors
Message-ID:
<CACWQX82Or8bnTA8WDqrogpp16vEff7PoEB4ZK4b3tFwYKWSQZQ@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"

This is great! Thank you for taking the time to type this up.

Quoted 2 lines · Alex Qiu

The setup has a total of 249 IPMI sensors, and among these, dbus-sensors reports 59 objects from HwmonTempSensor and 195 objects from PSUSensor, and we've already decreased the polling rate of PSUSensor to every 10 seconds to mitigate the issue. As the intel-ipmi-oem does, we measure the time of calling GetManagedObjects with commands:

This isn't the biggest sensor usage I've ever seen, but it certainly is the biggest usage of PSUsensor I've seen sofar. It's not surprising you're finding performance issues other people haven't. PSUSensor was originally supposed to be for physical pmbus power supplies, but got abstracted a little at some point to be more generic.

Quoted 6 lines

time busctl call xyz.openbmc_project.HwmonTempSensor / org.freedesktop.DBus.ObjectManager GetManagedObjects time busctl call xyz.openbmc_project.PSUSensor / org.freedesktop.DBus.ObjectManager GetManagedObjects

The first command for HwmonTempSensor constantly finishes in about 60 ms. However, the run time of the second command for PSUSensor is very inconsistent. Out of 50 continuous runs, most of them finish in about 150 ms, but 2 or 3 of them will take as long as about 6 seconds to return. This results in long time to scan the SDR and inconsistent performance polling IPMI sensors.

I don't have a system handy that uses PSUSensor, but based on what you're saying, I'm going to guess that there's a blocking io/wait/sleep call that snuck in somewhere in the PSUsensor, and it's stopping the main reactor for some amount of time. This is probably exacerbated by how loaded your system is, which is causing the really bad tail latencies.

If I were in your shoes, the first thing I would do is to recompile PSUSensor with IO handler tracking enabled:
https://www.boost.org/doc/libs/1_73_0/doc/html/boost_asio/overview/core/handler_tracking.html

to do that, go here:
https://github.com/openbmc/dbus-sensors/blob/master/CMakeLists.txt#L194

add add a line line like
target_compile_definitions(psusensor PUBLIC
-DBOOST_ASIO_ENABLE_HANDLER_TRACKING)

and recompile.

That's going to print loads of debug info to the console when it runs. Be prepared. Rerun your test with the flag enabled. When your getmanagedobjects command gets stuck, dump the log and try to find the spot where io seems to stop for a bit. Hopefully you'll find one async operation is taking a looooong time to run. Most operations should be in the order of micro/milliseconds for runtime. Once you know what the spot is, we can probably triage further. Each individual callback is pretty simple, and only does a couple things, so it should be pretty easy to sort out what's blocking within a given callback.

My second theory is that because of the async nature of psusensor, if you get unlucky, 195 concurrent IO completion operations are getting scheduled right ahead of your GetManagedObjects call. Right now the IO scheduling is pretty dumb, and doesn't attempt to add jitter to randomize the call starts, under the assumption that the reactor will never be more than 10 or so handles at a given time. Given the number of sensors you've got, we might want to rethink that, and try to spread them out in time a little. If we wanted to verify this, we could instrument io_context with a little run_for() magic that breaks every N milliseconds and prints the size of the queue. That could verify that we're running it too large.

Technically I think this is the embedded version of the thundering herd problem. There are ways to solve it that should be relatively easy (if that's what it turns out to be).

------------------------------

Message: 2
Date: Sat, 8 Aug 2020 01:19:03 +0000
From: Zhao Kun <[email protected]>
To: "Bills, Jason M" <[email protected]>,
"[email protected]" <[email protected]>
Subject: RE: system power control
Message-ID:

Quoted 6 lines
	<BYAPR14MB23424B7B0D6A450C52235EC2CF490@BYAPR14MB2342.namprd14.prod.outlook.com>

Content-Type: text/plain; charset="windows-1252"

Thank you, Jason. Could you share with me any example of defining those GPIOs in device tree for x86-power-control? I can?t find any in aspeed-bmc-intel-s2600wf.dts.

Please refer the below Facebook Tiogapass gpio device entry for x86-power-control.
https://github.com/openbmc/linux/blob/dev-5.7/arch/arm/boot/dts/aspeed-bmc-facebook-tiogapass.dts#L135

Quoted 67 lines

Thanks.

Best regards,

Kun Zhao
/*
[email protected]<mailto:[email protected]>
*/

From: Bills, Jason M<mailto:[email protected]>
Sent: Friday, August 7, 2020 10:12 AM
To: [email protected]<mailto:[email protected]>
Subject: Re: system power control

Quoted 13 lines · Zhao Kun

Hi,

I?m new to learn how to make OpenBMC work on a X86 based system. Currently I met a problem of mapping the GPIOs about power on/off/reset/status into OpenBMC logic. I understand when user issue a power on request through any user interfaces like RESTful, IPMI, etc., some service (phosphor-state-manager?) will be triggered to check current status and roll out corresponding systemd services to do the job. (please correct me if I?m wrong)

But I?m just confused on how those services actually toggle or check the GPIOs, there seems be many choices,

Quoted 8 lines

1. Device tree?
2. Using Workbook gpio_defs.json?
3. Create some services calling platform specific scripts to operate GPIO or I2C devices?
4. Using x86-power-control?

So what?s the most recommended way to do it? Really appreciated If anyone can share some lights.

On Intel reference platforms, we use x86-power-control and configure the GPIO names using device tree.

Quoted 9 lines

I thought there must be a mechanism to consume some kind of configuration file as the hardware abstraction layer. So I guess it might be gpio_defs.json or device tree.

Thanks.

Best regards,

Quoted 8 lines

Kun Zhao

/*

[email protected] <mailto:[email protected]>

*/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20200808/0244c0ef/attachment-0001.htm>

End of openbmc Digest, Vol 60, Issue 27
***************************************