Skip to contentSkip to conversation

dbus-sensors

DISCUSSION Matt Spinler 6 participants 11 messages Apr 21, 2020 – Jul 27, 2020

lore
Share & tools
lore
#0 lore raw

Hi James,

We're looking into using dbus-sensors(HwmonTemp and PSU) in the future, but would need to make a few changes to fit our requirements.  Was wondering what you'd think of the following:

1. Check if a sensor has a _fault sysfs file, and if it does and it is nonzero, set the Functional property on the OperationalStatus interface to false (and/or maybe 6 below).

2. After the 10 failed reads, instead of just setting the sensor to 0 also make a D-Bus call to create a phosphor-logging event log and set the OperationalStatus sensor to false.

3. After creating this event log, make sure not to do it again until main power is cycled.

4. If not already supported (was unsure), be able to find an _input file based on a value it has in the corresponding _label file.

5. We have a case where a driver isn't loaded with power off, so somehow we still need the sensors to stay on D-Bus when off (and show them as not available).

6. Maybe add a new property to Sensor.Value on the validity of the value property, for when driver is unloaded or there is an error or the sensor reading is otherwise not valid.  We could add this to phosphor-hwmon at the same time.
 (I think this was mentioned on the list before).

We would definitely of course work with you on the best way to accomplish these, and I know #6 needs more discussion on if this is something we want to do in OpenBMC, though I thought I remembered an earlier discussion where it was popular.

Thanks,
Matt

#1 lore raw
Quoted 9 lines · Matt Spinler

Hi James,

We're looking into using dbus-sensors(HwmonTemp and PSU) in the future, but would need to make a few changes to fit our requirements.  Was wondering what you'd think of the following:

1. Check if a sensor has a _fault sysfs file, and if it does and it is nonzero, set the Functional property on the OperationalStatus interface to false (and/or maybe 6 below)

Sounds ok.

Quoted 4 lines

2. After the 10 failed reads, instead of just setting the sensor to 0 also make a D-Bus call to create a phosphor-logging event log and set the OperationalStatus sensor to false.

Sounds ok.

Quoted 3 lines

3. After creating this event log, make sure not to do it again until main power is cycled.

I'd rather this be until the status goes OK again.

Quoted 3 lines

4. If not already supported (was unsure), be able to find an _input file based on a value it has in the corresponding _label file.

PSU sensor does this, hwmontemp does it by index.

Quoted 4 lines

5. We have a case where a driver isn't loaded with power off, so somehow we still need the sensors to stay on D-Bus when off (and show them as not available).

All sensors are on d-bus all the time, its based on the EM config.

Quoted 6 lines

6. Maybe add a new property to Sensor.Value on the validity of the value property, for when driver is unloaded or there is an error or the sensor reading is otherwise not valid.  We could add this to phosphor-hwmon at the same time.
 (I think this was mentioned on the list before).

Yes, this is where we've used std::nan, I'm not sure if that made it to all sensors as it's not tested very much. I know the fans do this.

Quoted 8 lines

We would definitely of course work with you on the best way to accomplish these, and I know #6 needs more discussion on if this is something we want to do in OpenBMC, though I thought I remembered an earlier discussion where it was popular.

Thanks,
Matt

#2 lore raw
Quoted 14 lines · James Feist
Quoted 7 lines · Matt Spinler

Hi James,
We're looking into using dbus-sensors(HwmonTemp and PSU) in the future, but would need to make a few changes to fit our requirements. Was wondering what you'd think of the following:
1. Check if a sensor has a _fault sysfs file, and if it does and it is nonzero, set the Functional property on the OperationalStatus interface to false (and/or maybe 6 below)

Sounds ok.

Quoted 3 lines

2. After the 10 failed reads, instead of just setting the sensor to 0 also make a D-Bus call to create a phosphor-logging event log and set the OperationalStatus sensor to false.

Quoted 6 lines

Sounds ok.

Quoted 2 lines

3. After creating this event log, make sure not to do it again until main power is cycled.

I'd rather this be until the status goes OK again.

We have user-experience requirements that the server administrator must be “nagged” in this fashion when something is broken like this. Could the behavior be selectable via build switch? Any other ideas on how to accommodate both behaviors?

Quoted 5 lines
Quoted 3 lines

4. If not already supported (was unsure), be able to find an _input file based on a value it has in the corresponding _label file.

PSU sensor does this, hwmontemp does it by index.

I think the question here was can we change the temp sensor implementation to do that also?

Quoted 20 lines
Quoted 3 lines

5. We have a case where a driver isn't loaded with power off, so somehow we still need the sensors to stay on D-Bus when off (and show them as not available).

All sensors are on d-bus all the time, its based on the EM config.

Quoted 5 lines

6. Maybe add a new property to Sensor.Value on the validity of the value property, for when driver is unloaded or there is an error or the sensor reading is otherwise not valid. We could add this to phosphor-hwmon at the same time.
(I think this was mentioned on the list before).

Yes, this is where we've used std::nan, I'm not sure if that made it to all sensors as it's not tested very much. I know the fans do this.

Quoted 6 lines

We would definitely of course work with you on the best way to accomplish these, and I know #6 needs more discussion on if this is something we want to do in OpenBMC, though I thought I remembered an earlier discussion where it was popular. Thanks,
Matt

#3 lore raw
Quoted 10 lines · Brad Bishop
Quoted 4 lines · James Feist
Quoted 2 lines

3. After creating this event log, make sure not to do it again until main power is cycled.

I'd rather this be until the status goes OK again.

We have user-experience requirements that the server administrator must be “nagged” in this fashion when something is broken like this. Could the behavior be selectable via build switch? Any other ideas on how to accommodate both behaviors?

This sounds like a form of error filtering. Shouldn't that decision be done at a much higher level in the stack than down in the entity that reads the hardware sensor?

Signature · 1 line
Patrick Williams
#4 lore raw
Quoted 7 lines · Patrick Williams
Quoted 6 lines · Brad Bishop
Quoted 4 lines · James Feist
Quoted 2 lines

3. After creating this event log, make sure not to do it again until main power is cycled.

I'd rather this be until the status goes OK again.

Quoted 8 lines
Quoted 4 lines

We have user-experience requirements that the server administrator must be “nagged” in this fashion when something is broken like this. Could the behavior be selectable via build switch? Any other ideas on how to accommodate both behaviors?

This sounds like a form of error filtering. Shouldn't that decision be done at a much higher level in the stack than down in the entity that reads the hardware sensor?

Thats an interesting thought. When the error reporting code sees the error for the first time, it could maintain a list of errors that it needs to “replay” at different system events, like when the server powers on.

This is certainly more flexible and I like the idea - but one down side though is the logging code becomes stateful and the complexity is slightly higher.

#5 lore raw
Quoted 14 lines · Brad Bishop
Quoted 12 lines · Patrick Williams
Quoted 10 lines · Brad Bishop
Quoted 4 lines · James Feist
Quoted 2 lines

3. After creating this event log, make sure not to do it again until main power is cycled.

I'd rather this be until the status goes OK again.

We have user-experience requirements that the server administrator must be “nagged” in this fashion when something is broken like this. Could the behavior be selectable via build switch?  Any other ideas on how to accommodate both behaviors?

Quoted 5 lines
Quoted 3 lines

This sounds like a form of error filtering.  Shouldn't that decision be done at a much higher level in the stack than down in the entity that reads the hardware sensor?

Thats an interesting thought.  When the error reporting code sees the error for the first time, it could maintain a list of errors that it needs to “replay” at different system events, like when the server powers on.

It isn't really nagging, it's more like error throttling.  At most, only log one error per power cycle. I have to check still, but we may also need to still log the other errors, just with a different severity (for debug purposes).

I kinda like this filtering idea too.  It is flexible and we would only have to do it in one place as opposed to in all the sensor applications we end up using, and could also be used to change the event log severities as mentioned above.  We will have to make sure when creating the event log that it contains enough information to recognize the device that is failing so that we can filter appropriately.

Quoted 2 lines

This is certainly more flexible and I like the idea - but one down side though is the logging code becomes stateful and the complexity is slightly higher.

#6 lore raw
Quoted 10 lines · Matt Spinler
Quoted 7 lines · Brad Bishop
Quoted 5 lines · Patrick Williams
Quoted 4 lines · Brad Bishop
Quoted 3 lines · James Feist
Quoted 2 lines

3. After creating this event log, make sure not to do it again until main power is cycled.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

I'd rather this be until the status goes OK again.

We have user-experience requirements that the server administrator must be “nagged” in this fashion when something is broken like this. Could the behavior be selectable via build switch?  Any other ideas on how to accommodate both behaviors?

As long as it's configurable somehow, fine by me.

Quoted 21 lines
Quoted 6 lines
Quoted 4 lines

This sounds like a form of error filtering.  Shouldn't that decision be done at a much higher level in the stack than down in the entity that reads the hardware sensor?

Thats an interesting thought.  When the error reporting code sees the error for the first time, it could maintain a list of errors that it needs to “replay” at different system events, like when the server powers on.

It isn't really nagging, it's more like error throttling.  At most, only log one error per power cycle. I have to check still, but we may also need to still log the other errors, just with a different severity (for debug purposes).

I kinda like this filtering idea too.  It is flexible and we would only have to do it in one place as opposed to in all the sensor applications we end up using, and could also be used to change the event log severities as mentioned above.  We will have to make sure when creating the event log that it contains enough information to recognize the device that is failing so that we can filter appropriately.

Quoted 2 lines

This is certainly more flexible and I like the idea - but one down side though is the logging code becomes stateful and the complexity is slightly higher.

#7 lore raw
In reply to James Feist

Great!  Sounds like we should be able to make thing work. A few comments below.

Quoted 17 lines · James Feist
Quoted 9 lines · Matt Spinler

Hi James,

We're looking into using dbus-sensors(HwmonTemp and PSU) in the future, but would need to make a few changes to fit our requirements. Was wondering what you'd think of the following:

1. Check if a sensor has a _fault sysfs file, and if it does and it is nonzero, set the Functional property on the OperationalStatus interface to false (and/or maybe 6 below)

Sounds ok.

Quoted 4 lines

2. After the 10 failed reads, instead of just setting the sensor to 0 also make a D-Bus call to create a phosphor-logging event log and set the OperationalStatus sensor to false.

Quoted 7 lines

Sounds ok.

Quoted 3 lines

3. After creating this event log, make sure not to do it again until main power is cycled.

I'd rather this be until the status goes OK again.

As suggested by Patrick, I agree the throttling can be done elsewhere, so we would just create the logs as you state here.

Quoted 6 lines
Quoted 4 lines

4. If not already supported (was unsure), be able to find an _input file based on a value it has in the corresponding _label file.

PSU sensor does this, hwmontemp does it by index.

Would you be OK with us also adding this to PSUSensor?

Quoted 6 lines
Quoted 4 lines

5. We have a case where a driver isn't loaded with power off, so somehow we still need the sensors to stay on D-Bus when off (and show them as not available).

All sensors are on d-bus all the time, its based on the EM config.

Perfect!

Quoted 18 lines
Quoted 7 lines

6. Maybe add a new property to Sensor.Value on the validity of the value property, for when driver is unloaded or there is an error or the sensor reading is otherwise not valid.  We could add this to phosphor-hwmon at the same time.
  (I think this was mentioned on the list before).

Yes, this is where we've used std::nan, I'm not sure if that made it to all sensors as it's not tested very much. I know the fans do this.

Quoted 8 lines

We would definitely of course work with you on the best way to accomplish these, and I know #6 needs more discussion on if this is something we want to do in OpenBMC, though I thought I remembered an earlier discussion where it was popular.

Thanks,
Matt

#8 lore raw
Quoted 24 lines · Matt Spinler

Great!  Sounds like we should be able to make thing work. A few comments below.

Quoted 20 lines · James Feist
Quoted 9 lines · Matt Spinler

Hi James,

We're looking into using dbus-sensors(HwmonTemp and PSU) in the future, but would need to make a few changes to fit our requirements. Was wondering what you'd think of the following:

1. Check if a sensor has a _fault sysfs file, and if it does and it is nonzero, set the Functional property on the OperationalStatus interface to false (and/or maybe 6 below)

Sounds ok.

Quoted 4 lines

2. After the 10 failed reads, instead of just setting the sensor to 0 also make a D-Bus call to create a phosphor-logging event log and set the OperationalStatus sensor to false.

Sounds ok.

Quoted 1 line
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

3. After creating this event log, make sure not to do it again until main power is cycled.

I'd rather this be until the status goes OK again.

As suggested by Patrick, I agree the throttling can be done elsewhere, so we would just create the logs as you state here.

I'm not familiar with the sensors, but for this specific case, would it work to base the log on OperationalStatus? It seems logical to not log events for sensors that are not operational, and since it will be set to false after the 10 failures, it would stop the log from nagging.

Quoted 37 lines
Quoted 6 lines
Quoted 4 lines

4. If not already supported (was unsure), be able to find an _input file based on a value it has in the corresponding _label file.

PSU sensor does this, hwmontemp does it by index.

Would you be OK with us also adding this to PSUSensor?

Quoted 6 lines
Quoted 4 lines

5. We have a case where a driver isn't loaded with power off, so somehow we still need the sensors to stay on D-Bus when off (and show them as not available).

All sensors are on d-bus all the time, its based on the EM config.

Perfect!

Quoted 18 lines
Quoted 7 lines

6. Maybe add a new property to Sensor.Value on the validity of the value property, for when driver is unloaded or there is an error or the sensor reading is otherwise not valid.  We could add this to phosphor-hwmon at the same time.
  (I think this was mentioned on the list before).

Yes, this is where we've used std::nan, I'm not sure if that made it to all sensors as it's not tested very much. I know the fans do this.

Quoted 8 lines

We would definitely of course work with you on the best way to accomplish these, and I know #6 needs more discussion on if this is something we want to do in OpenBMC, though I thought I remembered an earlier discussion where it was popular.

Thanks,
Matt

#9 lore raw
In reply to Matt Spinler
Quoted 7 lines · Matt Spinler
Quoted 5 lines
Quoted 3 lines

4. If not already supported (was unsure), be able to find an _input file based on a value it has in the corresponding _label file.

PSU sensor does this, hwmontemp does it by index.

Would you be OK with us also adding this to PSUSensor?

PSUSensor already does this? Do you mean hwmontemp sensor? I'd be fine with converting the index scheme to a label scheme.

#10 lore raw

Hi Jame:
I am investigating and porting psusensor to my platform. I have a
question item 5.

Quoted 5 lines
Quoted 3 lines

5. We have a case where a driver isn't loaded with power off, so somehow we still need the sensors to stay on D-Bus when off (and show them as not available).

All sensors are on d-bus all the time, its based on the EM config.

I create 2 psu config, psu1 and psu2, in entityManager and booting my system with 1 psu. There is only psu1 sensor object created in psusensor service.
After inserting psu2 and the driver binding hwmon for psu2, psusensor service don't create psu2 sensor object.
Does it work as design? or I need update dbus-sensor?
My dbus-sensor revision is "c140e2008b038b5fbf07aca0795b8030224eddd4".

Tyler

James Feist <[email protected]> 於 2020年4月23日 週四 上午1:27寫道:

Quoted 14 lines · Matt Spinler
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

4. If not already supported (was unsure), be able to find an _input file based on a value it has in the corresponding _label

file.

Quoted 4 lines
Quoted 2 lines

PSU sensor does this, hwmontemp does it by index.

Would you be OK with us also adding this to PSUSensor?

PSUSensor already does this? Do you mean hwmontemp sensor? I'd be fine with converting the index scheme to a label scheme.