RFC for Telemetry data collection
Hello,
I am working on the issue (https://github.com/openbmc/openbmc/issues/1957) to design a telemetry application for the OpenBMC. I would be explaining a rough idea of how we plan to go about. Please share your thoughts and feedback on this proposal. This issue would depend on the design evolving out of following issues, since this app would utilize the capabilities provided. (https://github.com/openbmc/openbmc/issues/1856, https://github.com/openbmc/openbmc/issues/2102).
Summary of the requirements that we came across relevant to this discussion.
1) BMC telemetry data (example VRM rail voltages) where the data is collected at different rates depending on the data and aggregated by the BMC app (minimum, maximum and average). Based on the collection timing request(frequency) the
metrics are logged, so that the user can fetch it for analytics.
2) Users should be able to set thresholds for the temperature limits, and receive alerts. This would allow user to plan the cooling needs.
3) BMC would act as route for the OCC metrics to be send to the user. The OCC would send down telemetric data to the BMC and BMC should figure out a way to alert the user to consume this data.
We would keep the focus of the discussion on the requirement no 1.
This proposal presupposes that all the resources( example VRM rail voltages, ambient temperature) that the telemetry app is interested in, should be populated as dbus objects, which can be queried to read the instantaneous values. phosphor-hwmon application exposes many of the interested resources.
The idea is to have a yaml based approach, where the policy of the telemetry app will be expressed. The application would be able to consume the yaml and initiate the telemetry data collection. The yaml would express the following:
a) Dbus Info (object, interface, property) associated with the resource.
b) Units associated with the value (celsius) and the associated scaling factor).
c) Granularity - the time between two measures.
d) Aggregation methods - min,max,avg..etc.
e) Logging policy - frequency for creating an event and alerting the user.
The application would operate based on the policy and log the telemetry data. The details of logging would evolve as we progress on the related issue.
Regards,
Tom
I have many opinions on telemetry data formats and APIs. What I'm seeing in your proposal looks pretty good with some subtlety in the details. For example, I expect to collect most data at least once-per-second, not log anything locally, and not alert. I'll do all aggregation and thresholding at a higher level in the software stack. I also, ideally, want very descriptive information about where in the system the sensor is. I've attached a screenshot of what our existing host-based reporting software makes available to higher-level software. This is a view via the human-readable web interface, the data is normally served via protobufs.
Rick
Quoted 59 lines · tomjose
Hello,
I am working on the issue (https://github.com/openbmc/openbmc/issues/1957)
to design a telemetry application for the OpenBMC. I would be explaining a rough idea of how we plan to go about. Please share your thoughts and feedback on this proposal. This issue would depend on the design evolving out of following issues, since this app would utilize the capabilities provided. (https://github.com/openbmc/openbmc/issues/1856,
https://github.com/openbmc/openbmc/issues/2102).Summary of the requirements that we came across relevant to this discussion.
1) BMC telemetry data (example VRM rail voltages) where the data is collected at different rates depending on the data and aggregated by the BMC app (minimum, maximum
and average). Based on the collection timing request(frequency) the
metrics are logged, so that the user can fetch it for analytics.2) Users should be able to set thresholds for the temperature limits, and receive alerts. This would allow user to plan the cooling needs.
3) BMC would act as route for the OCC metrics to be send to the user. The OCC would send down telemetric data to the BMC and BMC should figure out a way to
alert the user to consume this data.We would keep the focus of the discussion on the requirement no 1. This proposal presupposes that all the
resources( example VRM rail voltages, ambient temperature)that the telemetry app is interested in, should be populated as dbus objects, which can
be queried to read the instantaneous values. phosphor-hwmon application exposes many of the interested resources.The idea is to have a yaml based approach, where the policy of the telemetry app will be expressed. The application would be able to consume the yaml and initiate the telemetry
data collection. The yaml would express the following:a) Dbus Info (object, interface, property) associated with the resource.
b) Units associated with the value (celsius) and the associated scaling factor).
c) Granularity - the time between two measures.
d) Aggregation methods - min,max,avg..etc.
e) Logging policy - frequency for creating an event and alerting the user.The application would operate based on the policy and log the telemetry data. The details of logging would evolve as we progress on the related issue.
Regards,
Tom
We do need to provide live reads of a subset of the data, which I think is what Rick is describing below. For instance fan speeds, 30 second power averages, component temperatures, etc. Much like IPMI/DCMI implementations out there today. And these need to alert based on trip
levels that are set. Higher layers of software can then act on this data
or log it away as they see fit. I like the idea of rich meta-data around these values, but I would think we would use Redfish as the method of exporting this data.
We also need deep traces where the data is gathered, processed, and logged locally by the BMC. Then the BMC should alert every X hours and the log should be collected by the higher layer entity. This would be for things like VRM currents on every output (hourly min, max, average). It is not required that any other company use these deep telemetry logs, but they are required on our systems.
As far as #3 below, this should not be a new requirement. Just export the OCC telemetry log in the same way that you export all OCC/HOST logs.
Todd Rosedahl
IBM Power and Thermal Management
(507) 250-3275
[email protected]
From: Rick Altherr <[email protected]> To: tomjose <[email protected]> Cc: OpenBMC Maillist <[email protected]>, [email protected], [email protected], [email protected] Date: 09/07/2017 01:41 PM Subject: Re: RFC for Telemetry data collection
I have many opinions on telemetry data formats and APIs. What I'm seeing in your proposal looks pretty good with some subtlety in the details. For example, I expect to collect most data at least once-per-second, not log anything locally, and not alert. I'll do all aggregation and thresholding at a higher level in the software stack. I also, ideally, want very descriptive information about where in the system the sensor is. I've attached a screenshot of what our existing host-based reporting software makes available to higher-level software. This is a view via the human-readable web interface, the data is normally served via protobufs.
Rick
On Thu, Sep 7, 2017 at 8:20 AM, tomjose <[email protected]> wrote:
Hello,
I am working on the issue (https://github.com/openbmc/openbmc/issues/1957)
to design a telemetry application for the OpenBMC. I would be explaining a rough idea of how we plan to go about. Please share your thoughts and feedback on this proposal. This issue would depend on the design evolving out of following issues, since this app would utilize the capabilities provided. (https://github.com/openbmc/openbmc/issues/1856,
https://github.com/openbmc/openbmc/issues/2102).
Summary of the requirements that we came across relevant to this discussion.
1) BMC telemetry data (example VRM rail voltages) where the data is collected at different rates depending on the data and aggregated by the BMC app (minimum, maximum
and average). Based on the collection timing request(frequency) the
metrics are logged, so that the user can fetch it for analytics.
2) Users should be able to set thresholds for the temperature limits, and receive alerts. This would allow user to plan the cooling needs.
3) BMC would act as route for the OCC metrics to be send to the user. The OCC would send down telemetric data to the BMC and BMC should figure out a way to
alert the user to consume this data.
We would keep the focus of the discussion on the requirement no 1. This proposal presupposes that all the resources( example VRM rail voltages, ambient temperature) that the telemetry app is interested in, should be populated as dbus objects, which can
be queried to read the instantaneous values. phosphor-hwmon application exposes many of the interested resources.
The idea is to have a yaml based approach, where the policy of the telemetry app will be expressed. The application would be able to consume the yaml and initiate the telemetry
data collection. The yaml would express the following:
a) Dbus Info (object, interface, property) associated with the resource.
b) Units associated with the value (celsius) and the associated scaling factor).
c) Granularity - the time between two measures.
d) Aggregation methods - min,max,avg..etc.
e) Logging policy - frequency for creating an event and alerting the user.
The application would operate based on the policy and log the telemetry data. The details of logging would evolve as we progress on the related issue.
Regards,
Tom
[attachment "zaius-fan-telemetry.png" deleted by Todd Rosedahl/Rochester/IBM]
Quoted 4 lines · Rick Altherr
I have many opinions on telemetry data formats and APIs. What I'm seeing in your proposal looks pretty good with some subtlety in the details. For example, I expect to collect most data at least once-per-second,
This sounds like polling the BMC externally. Did I infer correctly? How do you feel about a push model with persistent connections?
Quoted 3 lines
not log anything locally, and not alert. I'll do all aggregation and thresholding at a higher level in the software stack. I also, ideally, want very descriptive information about where in the system the sensor is. I've attached a screenshot of what our existing host-based reporting software makes available to higher-level software. This is a view via the human-readable web interface, the data is normally served via protobufs.
Rick
Hi Tom
I don’t really disagree with anything you wrote…I’m just throwing out some additional thoughts.
I would propose that we:
1 - Add support to the REST server to subscribe to DBus signals, and simply forward the signal content in JSON format out over a websocket. This allows an external user to get any async notification that any code running on the BMC can get. I have a lot of questions on specifics here but I’ll save those for later, in case this doesn’t work out.
2 - Write an application to run on the BMC that subscribes to DBus signals and each time it occurs, creates a DBus object with the content of the signal again in JSON format. The signals to subscribe to (and therefore the resulting “history” DBus objects for) would be defined via per-platform YAML.
From there implementing different “eventing” interfaces like Redfish, DCMI, SNMP traps, or whatever Rick does with protobufs :-) becomes an exercise in:
1 - If the eventing interface is async, socket based: Doing the same as #1 above in a new application, but applying a transformation on the JSON. 2 - If the eventing interface is sync, history based: Write an application that transforms the objects created by #2 above.
Does anyone see this framework not enabling them to meet their requirements? Please poke holes.
-brad
Quoted 5 lines · tomjose
Hello,
I am working on the issue (https://github.com/openbmc/openbmc/issues/1957) to design a telemetry application for the OpenBMC. I would be explaining a rough idea of how we plan to go about. Please share your thoughts and feedback on this proposal.
Quoted 7 lines · Brad Bishop
Hi Tom
I don’t really disagree with anything you wrote…I’m just throwing out some additional thoughts.
I would propose that we:
1 - Add support to the REST server to subscribe to DBus signals, and simply forward the signal content in JSON format out over a websocket. This allows an external user to get any async notification that any code running on the BMC can get. I have a lot of questions on specifics here but I’ll save those for later, in case this doesn’t work out.
Brad, that sounds fine to me in terms of how a notification can be sent, although Tom's proposal also talked about describing in yaml, how frequently to read something off of a, say a d-bus object representing a sensor. If I understood correctly, are you saying instead of doing this every X hours or seconds, we send out a notification whenever a PropertyChanged d-bus signal is caught? And let an off-BMC application bother about averages, min/max?
Regards,
Deepak
Quoted 9 lines · Deepak Kodihalli
Quoted 4 lines · Brad Bishop
Hi Tom
I don’t really disagree with anything you wrote…I’m just throwing out some additional thoughts.
I would propose that we:
1 - Add support to the REST server to subscribe to DBus signals, and simply forward the signal content in JSON format out over a websocket. This allows an external user to get any async notification that any code running on the BMC can get. I have a lot of questions on specifics here but I’ll save those for later, in case this doesn’t work out.
You only quoted #1 here. I’m highlighting that because terms like notification, event, and telemetry are being used a little loosely and we are not all on the same page.
I think we need to accommodate both async and sync/historical modes where async is just streamed out of the BMC, and sync, out of necessity have dbus objects created. Hence my proposal for two applications, one for each mode.
Really trying to drive my point here - to put the above another way - in terms of how I have described these two modes, Toms thread (this thread) would be tackling async mode and your note thread (https://lists.ozlabs.org/pipermail/openbmc/2017-September/009065.html) covering sync/historical mode.
Quoted 1 line
Brad, that sounds fine to me in terms of how a notification can be sent, although Tom's proposal also talked about describing in yaml, how frequently to read something off of a, say a d-bus object representing a sensor.
My initial thinking is that neither of the proposed applications would ever be doing sync reads to a dbus object. Rather, they just passively sit there and consume signals. If need be they could discard them (see rate-limiting below).
Quoted 1 line
If I understood correctly, are you saying instead of doing this every X hours or seconds, we send out a notification whenever a PropertyChanged d-bus signal is caught?
Yes and no. I am proposing we consume signals rather than perform reads but I still think we need:
1 - An API to configure async mode (what to subscribe to, maybe some kind of rate limiting). I’m not sure if this would be a DBus API or just a REST thing.
2 - A robust YAML syntax that instructs application #2 (sync mode) to do the same things as #1 (what to subscribe to, rate limiting when creating historical snapshot objects).
Quoted 5 lines
And let an off-BMC application bother about averages, min/max?
Regards,
Deepak
Quoted 30 lines · tomjose
Hello,
I am working on the issue (https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openbmc_openbmc_issues_1957&d=DwICaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=LzkOghL3x_3V_EkHiUQJUJ0xrq_s3_wwfssFT35AQXw&m=Gk0KnGVKy2iC82jVgpSqjzR2K_EYRlsFBqs34EjyKys&s=nQguk1LTD_q7dPNEn6dV2p4vsKaMu1vYZs5_Vh0BiNc&e= ) to design a telemetry application for the OpenBMC. I would be explaining a rough idea of how we plan to go about. Please share your thoughts and feedback on this proposal. This issue would depend on the design evolving out of following issues, since this app would utilize the capabilities provided. (https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openbmc_openbmc_issues_1856&d=DwICaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=LzkOghL3x_3V_EkHiUQJUJ0xrq_s3_wwfssFT35AQXw&m=Gk0KnGVKy2iC82jVgpSqjzR2K_EYRlsFBqs34EjyKys&s=2B_nLYU03S0QgMnCrMr8YawangOxRXmXGBqPF9593DY&e= , https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openbmc_openbmc_issues_2102&d=DwICaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=LzkOghL3x_3V_EkHiUQJUJ0xrq_s3_wwfssFT35AQXw&m=Gk0KnGVKy2iC82jVgpSqjzR2K_EYRlsFBqs34EjyKys&s=U6M9vpoDDmNbTJJH5I6M6lPBGFzS1nuqYEEGwXjAviY&e= ).
Summary of the requirements that we came across relevant to this discussion.
1) BMC telemetry data (example VRM rail voltages) where the data is collected at different rates depending on the data and aggregated by the BMC app (minimum, maximum and average). Based on the collection timing request(frequency) the
metrics are logged, so that the user can fetch it for analytics.2) Users should be able to set thresholds for the temperature limits, and receive alerts. This would allow user to plan the cooling needs.
3) BMC would act as route for the OCC metrics to be send to the user. The OCC would send down telemetric data to the BMC and BMC should figure out a way to
alert the user to consume this data.We would keep the focus of the discussion on the requirement no 1.
This proposal presupposes that all theresources( example VRM rail voltages, ambient temperature)that the telemetry app is interested in, should be populated as dbus objects, which can be queried to read the instantaneous values. phosphor-hwmon application exposes many of the interested resources.The idea is to have a yaml based approach, where the policy of the telemetry app will be expressed. The application would be able to consume the yaml and initiate the telemetry data collection. The yaml would express the following:
a) Dbus Info (object, interface, property) associated with the resource.
b) Units associated with the value (celsius) and the associated scaling factor).
c) Granularity - the time between two measures.
d) Aggregation methods - min,max,avg..etc.
e) Logging policy - frequency for creating an event and alerting the user.
Quoted 4 lines
The application would operate based on the policy and log the telemetry data. The details of logging would evolve as we progress on the related issue.
Regards,
Tom
Hi,
I'd like to bump this topic and add some more details. I'd like to discuss design proposals/directions for a couple things :
1) A short/mid term proposal for telemetry requirements specific to IBM labs (which need to be implemented in a relatively short span of time, so there may not be the bandwidth to write an entirely new application not based on D-Bus or the OpenBMC REST API).
2) Industry standard methods for storing and retrieving telemetry data - thoughts on how to get here.
1) Telemetry requirements specific to IBM labs
Here are the requirements and a design proposal.
a) Instantaneous readings, such as temperatures, currents, errors, events etc. Let's call this Layer 0.
Proposal:
- The D-Bus model is the source for instantaneous readings. This means there would be D-Bus objects representing this data, and hence an OpenBMC REST API around it.
- These D-Bus objects would not necessarily implement the same D-Bus interfaces.
- Interested clients can read these D-Bus objects via the OpenBMC REST API.
- If clients are interested in being notified about "changes" to the readings, that's possible via the existing event notification over WebSockets mechanism.
b) Instantaneous aggregations - this would mostly apply to, but may not be limited to, readings such as temperatures and currents. Let's call this Layer 1. This basically is to solve, for eg, "what is the min/max/average over the last X seconds?". We have a requirement to do such aggregations on the BMC.
Proposal:
- Aggregations are represented as D-Bus objects, created by a telemetry app. For eg if we need to know the min/max/avg ambient temp for the last 5 minutes, and say the the ambient temp is usually at temps/ambient, the aggregation could be at temps/aggregations/ambient.
- Implement D-Bus interfaces to denote aggregations, for eg the temps/aggregation/ambient object could implement a D-Bus interface describing min/max/avg properties.
- Aggregation objects will have the values as described in the D-Bus interface (such as min/max/avg), and a timestamp, as properties.
- Enable a config (eg JSON) to let the telemetry app know things like : What (supported) aggregations should be performed (min/max/avg)? What D-Bus objects should be aggregated? How frequently should they be aggregated? What should be the paths of the aggregations? Potentially add a REST API to allow changing the (JSON) config at run-time.
- It will be possible to read all aggregation objects, or aggregation objects of a specific type via one REST call.
c) Historical aggregations or snapshot. Let's call this Layer 2. This is to solve, for eg, "Need a reading corresponding to every X minutes in a period of Y hours". This can be a snapshot of Layer 1 or Layer 0 D-Bus objects. We have a requirement to store this snapshot on the BMC.
Proposal:
- The snapshot will be represented as a set of D-Bus objects. For eg if one needs an hourly reading for a period of 24 hours, the objects could be at temps/aggregations/ambient/per-hour/{1..24}.
- Enable a config to let a telemetry app to know things like : What D-Bus objects should I keep a history of? What is the duration of the snapshot? At what frequency should entries be added into the snapshot? Once the snapshot is full, should the entries roll, or should we restart? Potentially add a REST API to allow changing the (JSON) config at run-time.
- The historical aggregations can be read via one REST call. It should be one D-Bus call as well most likely for the REST server, if there's an object manager at temps/aggregations/ambient/per-hour for eg.
- These objects in the snapshot will implement the same interfaces as Layer 1 objects, so they will have the same properties (eg min/max/avg, timestamp).
d) Some notes
- With the proposal above, the API to retrieve the telemetry data is via the current OpenBMC REST API, so it may not readily work with telemetry tools relying on industry-standard API (see point 2 below), but it seems to be the feasible option to rely on to implement IBM's requirements in the expected timelines.
- Layer 1 and Layer 2 telemetry apps would be different processes, and can function independent of each other.
2) Industry standard methods for storing and retrieving telemetry data
- With the proposal above, the instantaneous readings are D-Bus objects, the instantaneous and historical aggregations are D-Bus objects as well. The API is the OpenBMC REST API.
- Typically, aggregations may not have to happen on the BMC, in which case one can turn off layers 1 and 2.
- This is regarding how the telemetry data is presented, and how we'd eventually not use the current OpenBMC REST API in production. I've heard (mostly from people on the To: list) of the following industry-standard ways to represent/retrieve telemetry data. This would mean transforming layer 0 D-Bus objects into these :
- Via Redfish (events) API
- Via IPMI events/PEF
- Via SNMP traps
- Via an sqlite db, and have something like Logstash parse it
- Others?
Discussions are already happening regarding Redfish, so telemetry could be one aspect to consider as well.
- Aggregations could be done on the BMC with collectd. I need to look at this in detail. Aggregations could be stored in an RRD format. Need to look at this in detail as well. These are as opposed to a D-Bus model of aggregations. Thoughts on this? For eg, would this be much less work both for the BMC and the telemetry data users than the proposed D-Bus model, but at the same time can address the requirements I've mentioned? Do we know what are the commonly used client tools for processing telemetry data, and how they expect the data to be presented?
Thanks,
Deepak
Quoted 114 lines · Deepak Kodihalli
Quoted 29 lines · tomjose
Hello,
I am working on the issue (https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openbmc_openbmc_issues_1957&d=DwICaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=LzkOghL3x_3V_EkHiUQJUJ0xrq_s3_wwfssFT35AQXw&m=Gk0KnGVKy2iC82jVgpSqjzR2K_EYRlsFBqs34EjyKys&s=nQguk1LTD_q7dPNEn6dV2p4vsKaMu1vYZs5_Vh0BiNc&e= ) to design a telemetry application for the OpenBMC. I would be explaining a rough idea of how we plan to go about. Please share your thoughts and feedback on this proposal. This issue would depend on the design evolving out of following issues, since this app would utilize the capabilities provided. (https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openbmc_openbmc_issues_1856&d=DwICaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=LzkOghL3x_3V_EkHiUQJUJ0xrq_s3_wwfssFT35AQXw&m=Gk0KnGVKy2iC82jVgpSqjzR2K_EYRlsFBqs34EjyKys&s=2B_nLYU03S0QgMnCrMr8YawangOxRXmXGBqPF9593DY&e= , https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openbmc_openbmc_issues_2102&d=DwICaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=LzkOghL3x_3V_EkHiUQJUJ0xrq_s3_wwfssFT35AQXw&m=Gk0KnGVKy2iC82jVgpSqjzR2K_EYRlsFBqs34EjyKys&s=U6M9vpoDDmNbTJJH5I6M6lPBGFzS1nuqYEEGwXjAviY&e= ).
Summary of the requirements that we came across relevant to this discussion.
1) BMC telemetry data (example VRM rail voltages) where the data is collected at different rates depending on the data and aggregated by the BMC app (minimum, maximum and average). Based on the collection timing request(frequency) the metrics are logged, so that the user can fetch it for analytics.
2) Users should be able to set thresholds for the temperature limits, and receive alerts. This would allow user to plan the cooling needs.
3) BMC would act as route for the OCC metrics to be send to the user. The OCC would send down telemetric data to the BMC and BMC should figure out a way to alert the user to consume this data.
We would keep the focus of the discussion on the requirement no 1.
This proposal presupposes that all theresources( example VRM rail voltages, ambient temperature)that the telemetry app is interested in, should be populated as dbus objects, which can be queried to read the instantaneous values. phosphor-hwmon application exposes many of the interested resources.The idea is to have a yaml based approach, where the policy of the telemetry app will be expressed. The application would be able to consume the yaml and initiate the telemetry data collection. The yaml would express the following:
a) Dbus Info (object, interface, property) associated with the resource.
b) Units associated with the value (celsius) and the associated scaling factor).
c) Granularity - the time between two measures.
d) Aggregation methods - min,max,avg..etc.
e) Logging policy - frequency for creating an event and alerting the user.Quoted 4 lines
The application would operate based on the policy and log the telemetry data. The details of logging would evolve as we progress on the related issue.
Regards,
TomHi,
I'd like to bump this topic and add some more details. I'd like to discuss design proposals/directions for a couple things :
1) A short/mid term proposal for telemetry requirements specific to IBM labs (which need to be implemented in a relatively short span of time, so there may not be the bandwidth to write an entirely new application not based on D-Bus or the OpenBMC REST API).
2) Industry standard methods for storing and retrieving telemetry data - thoughts on how to get here.1) Telemetry requirements specific to IBM labs
Here are the requirements and a design proposal.a) Instantaneous readings, such as temperatures, currents, errors, events etc. Let's call this Layer 0.
Proposal:
- The D-Bus model is the source for instantaneous readings. This means
there would be D-Bus objects representing this data, and hence an OpenBMC REST API around it.
- These D-Bus objects would not necessarily implement the same D-Bus interfaces.
- Interested clients can read these D-Bus objects via the OpenBMC REST API.
- If clients are interested in being notified about "changes" to the
readings, that's possible via the existing event notification over WebSockets mechanism.b) Instantaneous aggregations - this would mostly apply to, but may not be limited to, readings such as temperatures and currents. Let's call this Layer 1. This basically is to solve, for eg, "what is the min/max/average over the last X seconds?". We have a requirement to do such aggregations on the BMC.
Proposal:
- Aggregations are represented as D-Bus objects, created by a telemetry
app. For eg if we need to know the min/max/avg ambient temp for the last 5 minutes, and say the the ambient temp is usually at temps/ambient, the aggregation could be at temps/aggregations/ambient.
- Implement D-Bus interfaces to denote aggregations, for eg the
temps/aggregation/ambient object could implement a D-Bus interface describing min/max/avg properties.
- Aggregation objects will have the values as described in the D-Bus
interface (such as min/max/avg), and a timestamp, as properties.
- Enable a config (eg JSON) to let the telemetry app know things like :
What (supported) aggregations should be performed (min/max/avg)? What D-Bus objects should be aggregated? How frequently should they be aggregated? What should be the paths of the aggregations? Potentially add a REST API to allow changing the (JSON) config at run-time.
- It will be possible to read all aggregation objects, or aggregation
objects of a specific type via one REST call.c) Historical aggregations or snapshot. Let's call this Layer 2. This is to solve, for eg, "Need a reading corresponding to every X minutes in a period of Y hours". This can be a snapshot of Layer 1 or Layer 0 D-Bus objects. We have a requirement to store this snapshot on the BMC.
Proposal:
- The snapshot will be represented as a set of D-Bus objects. For eg if
one needs an hourly reading for a period of 24 hours, the objects could be at temps/aggregations/ambient/per-hour/{1..24}.
- Enable a config to let a telemetry app to know things like : What
D-Bus objects should I keep a history of? What is the duration of the snapshot? At what frequency should entries be added into the snapshot? Once the snapshot is full, should the entries roll, or should we restart? Potentially add a REST API to allow changing the (JSON) config at run-time.
- The historical aggregations can be read via one REST call. It should
be one D-Bus call as well most likely for the REST server, if there's an object manager at temps/aggregations/ambient/per-hour for eg.
- These objects in the snapshot will implement the same interfaces as
Layer 1 objects, so they will have the same properties (eg min/max/avg, timestamp).d) Some notes
- With the proposal above, the API to retrieve the telemetry data is via
the current OpenBMC REST API, so it may not readily work with telemetry tools relying on industry-standard API (see point 2 below), but it seems to be the feasible option to rely on to implement IBM's requirements in the expected timelines.
- Layer 1 and Layer 2 telemetry apps would be different processes, and
can function independent of each other.2) Industry standard methods for storing and retrieving telemetry data
- With the proposal above, the instantaneous readings are D-Bus objects,
the instantaneous and historical aggregations are D-Bus objects as well. The API is the OpenBMC REST API.
- Typically, aggregations may not have to happen on the BMC, in which
case one can turn off layers 1 and 2.
- This is regarding how the telemetry data is presented, and how we'd
eventually not use the current OpenBMC REST API in production. I've heard (mostly from people on the To: list) of the following industry-standard ways to represent/retrieve telemetry data. This would mean transforming layer 0 D-Bus objects into these :
- Via Redfish (events) API
- Via IPMI events/PEF
- Via SNMP traps
- Via an sqlite db, and have something like Logstash parse it
- Others?
Discussions are already happening regarding Redfish, so telemetry could be one aspect to consider as well.
- Aggregations could be done on the BMC with collectd. I need to look at
this in detail. Aggregations could be stored in an RRD format. Need to look at this in detail as well. These are as opposed to a D-Bus model of aggregations. Thoughts on this? For eg, would this be much less work both for the BMC and the telemetry data users than the proposed D-Bus model, but at the same time can address the requirements I've mentioned? Do we know what are the commonly used client tools for processing telemetry data, and how they expect the data to be presented?
Quoted 2 lines
Thanks,
Deepak
Added this topic to the agenda in the wiki. Would appreciate feedback so as to talk about the same in the telecon.
Thanks,
Deepak
Quoted 87 lines · Deepak Kodihalli
Quoted 58 lines · tomjose
Hello,
I am working on the issue (https://urldefense.proofpoint
.com/v2/url?u=https-3A__github.com_openbmc_openbmc_ issues_1957&d=DwICaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=LzkOghL3 x_3V_EkHiUQJUJ0xrq_s3_wwfssFT35AQXw&m=Gk0KnGVKy2iC82jVgpSqjz R2K_EYRlsFBqs34EjyKys&s=nQguk1LTD_q7dPNEn6dV2p4vsKaMu1vYZs5_Vh0BiNc&e= ) to design a telemetry application for the OpenBMC. I would be explaining a rough idea of how we plan to go about. Please share your thoughts and feedback on this proposal. This issue would depend on the design evolving out of following issues, since this app would utilize the capabilities provided. (https://urldefense.proofpoint.com/v2/url?u=https-3A__
github.com_openbmc_openbmc_issues_1856&d=DwICaQ&c=jf_ iaSHvJObTbx-siA1ZOg&r=LzkOghL3x_3V_EkHiUQJUJ0xrq_s3_wwfssFT3 5AQXw&m=Gk0KnGVKy2iC82jVgpSqjzR2K_EYRlsFBqs34EjyKys&s=2B_nLY U03S0QgMnCrMr8YawangOxRXmXGBqPF9593DY&e= , https://urldefense.proofpoint.
com/v2/url?u=https-3A__github.com_openbmc_openbmc_issues_210 2&d=DwICaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=LzkOghL3x_3V_ EkHiUQJUJ0xrq_s3_wwfssFT35AQXw&m=Gk0KnGVKy2iC82jVgpSqjzR2K_E YRlsFBqs34EjyKys&s=U6M9vpoDDmNbTJJH5I6M6lPBGFzS1nuqYEEGwXjAviY&e= ).Summary of the requirements that we came across relevant to this discussion.
1) BMC telemetry data (example VRM rail voltages) where the data is collected at different rates depending on the data and aggregated by the BMC app (minimum, maximum
and average). Based on the collection timing request(frequency) the
metrics are logged, so that the user can fetch it for analytics.2) Users should be able to set thresholds for the temperature limits, and receive alerts. This would allow user to plan the cooling needs.
3) BMC would act as route for the OCC metrics to be send to the user. The OCC would send down telemetric data to the BMC and BMC should figure out a way to
alert the user to consume this data.We would keep the focus of the discussion on the requirement no 1. This proposal presupposes that all the
resources( example VRM rail voltages, ambient temperature)that the telemetry app is interested in, should be populated as dbus objects, which can
be queried to read the instantaneous values. phosphor-hwmon application exposes many of the interested resources.The idea is to have a yaml based approach, where the policy of the telemetry app will be expressed. The application would be able to consume the yaml and initiate the telemetry
data collection. The yaml would express the following:a) Dbus Info (object, interface, property) associated with the resource.
b) Units associated with the value (celsius) and the associated scaling factor).
c) Granularity - the time between two measures.
d) Aggregation methods - min,max,avg..etc.
e) Logging policy - frequency for creating an event and alerting the user.Quoted 7 lines
The application would operate based on the policy and log the telemetry data. The details of logging would evolve as we progress on the related issue.
Regards,
TomHi,
I'd like to bump this topic and add some more details. I'd like to discuss design proposals/directions for a couple things :
1) A short/mid term proposal for telemetry requirements specific to IBM labs (which need to be implemented in a relatively short span of time, so there may not be the bandwidth to write an entirely new application not based on D-Bus or the OpenBMC REST API).
2) Industry standard methods for storing and retrieving telemetry data - thoughts on how to get here.1) Telemetry requirements specific to IBM labs
Here are the requirements and a design proposal.a) Instantaneous readings, such as temperatures, currents, errors, events etc. Let's call this Layer 0.
Quoted 11 lines
Proposal:
- The D-Bus model is the source for instantaneous readings. This means there would be D-Bus objects representing this data, and hence an OpenBMC REST API around it.
- These D-Bus objects would not necessarily implement the same D-Bus interfaces.
- Interested clients can read these D-Bus objects via the OpenBMC REST API.
- If clients are interested in being notified about "changes" to the readings, that's possible via the existing event notification over WebSockets mechanism.
This would also map well into an OBMC MIB extension for example.
Quoted 8 lines
b) Instantaneous aggregations - this would mostly apply to, but may not be limited to, readings such as temperatures and currents. Let's call this Layer 1. This basically is to solve, for eg, "what is the min/max/average over the last X seconds?". We have a requirement to do such aggregations on the BMC.
I would be interested in why aggregations (and historical - level 3) are a requirement and not just handled by the monitoring/event management app as done in network management. If this work is to be done in the BMC, it needs to be user definable and able to be turned off for resource-critical situations.
Quoted 55 lines
Proposal:
- Aggregations are represented as D-Bus objects, created by a telemetry app. For eg if we need to know the min/max/avg ambient temp for the last 5 minutes, and say the the ambient temp is usually at temps/ambient, the aggregation could be at temps/aggregations/ambient.
- Implement D-Bus interfaces to denote aggregations, for eg the temps/aggregation/ambient object could implement a D-Bus interface describing min/max/avg properties.
- Aggregation objects will have the values as described in the D-Bus interface (such as min/max/avg), and a timestamp, as properties.
- Enable a config (eg JSON) to let the telemetry app know things like :
What (supported) aggregations should be performed (min/max/avg)? What D-Bus objects should be aggregated? How frequently should they be aggregated? What should be the paths of the aggregations? Potentially add a REST API to allow changing the (JSON) config at run-time.
- It will be possible to read all aggregation objects, or aggregation objects of a specific type via one REST call.c) Historical aggregations or snapshot. Let's call this Layer 2. This is to solve, for eg, "Need a reading corresponding to every X minutes in a period of Y hours". This can be a snapshot of Layer 1 or Layer 0 D-Bus objects. We have a requirement to store this snapshot on the BMC.
Proposal:
- The snapshot will be represented as a set of D-Bus objects. For eg if one needs an hourly reading for a period of 24 hours, the objects could be at temps/aggregations/ambient/per-hour/{1..24}.
- Enable a config to let a telemetry app to know things like : What D-Bus objects should I keep a history of? What is the duration of the snapshot? At what frequency should entries be added into the snapshot? Once the snapshot is full, should the entries roll, or should we restart? Potentially add a REST API to allow changing the (JSON) config at run-time.
- The historical aggregations can be read via one REST call. It should be one D-Bus call as well most likely for the REST server, if there's an object manager at temps/aggregations/ambient/per-hour for eg.
- These objects in the snapshot will implement the same interfaces as Layer 1 objects, so they will have the same properties (eg min/max/avg, timestamp).d) Some notes
- With the proposal above, the API to retrieve the telemetry data is via the current OpenBMC REST API, so it may not readily work with telemetry tools relying on industry-standard API (see point 2 below), but it seems to be the feasible option to rely on to implement IBM's requirements in the expected timelines.
- Layer 1 and Layer 2 telemetry apps would be different processes, and can function independent of each other.2) Industry standard methods for storing and retrieving telemetry data
Quoted 13 lines
- With the proposal above, the instantaneous readings are D-Bus objects, the instantaneous and historical aggregations are D-Bus objects as well. The API is the OpenBMC REST API.
- Typically, aggregations may not have to happen on the BMC, in which case one can turn off layers 1 and 2.
- This is regarding how the telemetry data is presented, and how we'd eventually not use the current OpenBMC REST API in production. I've heard (mostly from people on the To: list) of the following industry-standard ways to represent/retrieve telemetry data. This would mean transforming layer 0 D-Bus objects into these :
- Via Redfish (events) API
- Via IPMI events/PEF
Meh. I'd stick with Redfish/OBMC REST API over this one.
Quoted 2 lines
- Via SNMP traps
If there is interest here, I have experience designing MIB extensions and sub-agents to support them.
Quoted 2 lines
- Via an sqlite db, and have something like Logstash parse it
Seems very heavy for BMC.
Quoted 14 lines
- Others?
Discussions are already happening regarding Redfish, so telemetry could be one aspect to consider as well.
- Aggregations could be done on the BMC with collectd. I need to look at this in detail. Aggregations could be stored in an RRD format. Need to look at this in detail as well. These are as opposed to a D-Bus model of aggregations. Thoughts on this? For eg, would this be much less work both for the BMC and the telemetry data users than the proposed D-Bus model, but at the same time can address the requirements I've mentioned? Do we know what are the commonly used client tools for processing telemetry data, and how they expect the data to be presented?
Quoted 4 lines
Thanks,
Deepak
Kurt Taylor (krtaylor)
Quoted 36 lines · Kurt Taylor
Hi,
I'd like to bump this topic and add some more details. I'd like to discuss design proposals/directions for a couple things :
1) A short/mid term proposal for telemetry requirements specific to IBM labs (which need to be implemented in a relatively short span of time, so there may not be the bandwidth to write an entirely new application not based on D-Bus or the OpenBMC REST API).
2) Industry standard methods for storing and retrieving telemetry data - thoughts on how to get here.1) Telemetry requirements specific to IBM labs Here are the requirements and a design proposal.
a) Instantaneous readings, such as temperatures, currents, errors, events etc. Let's call this Layer 0.
Proposal:
- The D-Bus model is the source for instantaneous readings. This means there would be D-Bus objects representing this data, and hence an OpenBMC REST API around it.
- These D-Bus objects would not necessarily implement the same D-Bus interfaces.
- Interested clients can read these D-Bus objects via the OpenBMC REST API.
- If clients are interested in being notified about "changes" to the readings, that's possible via the existing event notification over WebSockets mechanism.This would also map well into an OBMC MIB extension for example.
Quoted 8 lines
b) Instantaneous aggregations - this would mostly apply to, but may not be limited to, readings such as temperatures and currents. Let's call this Layer 1. This basically is to solve, for eg, "what is the min/max/average over the last X seconds?". We have a requirement to do such aggregations on the BMC.
I would be interested in why aggregations (and historical - level 3) are a requirement and not just handled by the monitoring/event management app as done in network management.If this work is to be done in the BMC, it needs to be user definable and able to be turned off for resource-critical situations.
Right, it should be possible to turn off the layer 2 and 3 aggregation apps, and not have them in the BMC image at all.
Why the aggregations are required to be done on the BMC - I think that's the expectation of some of the IBM monitoring tools. I'm sure Todd Rosedahl would have a better answer here.
Quoted 78 lines
Proposal:
- Aggregations are represented as D-Bus objects, created by a telemetry app. For eg if we need to know the min/max/avg ambient temp for the last 5 minutes, and say the the ambient temp is usually at temps/ambient, the aggregation could be at temps/aggregations/ambient.
- Implement D-Bus interfaces to denote aggregations, for eg the temps/aggregation/ambient object could implement a D-Bus interface describing min/max/avg properties.
- Aggregation objects will have the values as described in the D-Bus interface (such as min/max/avg), and a timestamp, as properties.
- Enable a config (eg JSON) to let the telemetry app know things like : What (supported) aggregations should be performed (min/max/avg)? What D-Bus objects should be aggregated? How frequently should they be aggregated? What should be the paths of the aggregations? Potentially add a REST API to allow changing the (JSON) config at run-time.
- It will be possible to read all aggregation objects, or aggregation objects of a specific type via one REST call.c) Historical aggregations or snapshot. Let's call this Layer 2. This is to solve, for eg, "Need a reading corresponding to every X minutes in a period of Y hours". This can be a snapshot of Layer 1 or Layer 0 D-Bus objects. We have a requirement to store this snapshot on the BMC.
Proposal:
- The snapshot will be represented as a set of D-Bus objects. For eg if one needs an hourly reading for a period of 24 hours, the objects could be at temps/aggregations/ambient/per-hour/{1..24}.
- Enable a config to let a telemetry app to know things like : What D-Bus objects should I keep a history of? What is the duration of the snapshot? At what frequency should entries be added into the snapshot? Once the snapshot is full, should the entries roll, or should we restart? Potentially add a REST API to allow changing the (JSON) config at run-time.
- The historical aggregations can be read via one REST call. It should be one D-Bus call as well most likely for the REST server, if there's an object manager at temps/aggregations/ambient/per-hour for eg.
- These objects in the snapshot will implement the same interfaces as Layer 1 objects, so they will have the same properties (eg min/max/avg, timestamp).d) Some notes
- With the proposal above, the API to retrieve the telemetry data is via the current OpenBMC REST API, so it may not readily work with telemetry tools relying on industry-standard API (see point 2 below), but it seems to be the feasible option to rely on to implement IBM's requirements in the expected timelines.
- Layer 1 and Layer 2 telemetry apps would be different processes, and can function independent of each other.2) Industry standard methods for storing and retrieving telemetry data
- With the proposal above, the instantaneous readings are D-Bus objects, the instantaneous and historical aggregations are D-Bus objects as well. The API is the OpenBMC REST API.
- Typically, aggregations may not have to happen on the BMC, in which case one can turn off layers 1 and 2.
- This is regarding how the telemetry data is presented, and how we'd eventually not use the current OpenBMC REST API in production. I've heard (mostly from people on the To: list) of the following industry-standard ways to represent/retrieve telemetry data. This would mean transforming layer 0 D-Bus objects into these :
- Via Redfish (events) API
- Via IPMI events/PEFMeh. I'd stick with Redfish/OBMC REST API over this one.
- Via SNMP traps
Quoted 6 lines
If there is interest here, I have experience designing MIB extensions and sub-agents to support them.
- Via an sqlite db, and have something like Logstash parse it
Seems very heavy for BMC.
I tend to agree.
Quoted 2 lines
Kurt Taylor (krtaylor)
Regards,
Deepak