A question about phosphor-sel-logger
Hi,
Our BMC is using package "phosphor-sel-logger" to support IPMI SEL. it saved the event data in the journal and the id is 1-65534.
It means it can save 65534 records. Actually, we found the file size of single journal file is around 8MB(the default of journal file is 64MB, it will be separated by 8 files). We cannot show all SELs(65534 records) in IPMI. it might be a bug here if we don't limit the maximum.
Is it necessary to limit the number of SELs to smaller?
BRs,
Will
Quoted 7 lines · Will Liang (梁永鉉)
Hi,
Our BMC is using package "phosphor-sel-logger" to support IPMI SEL. it saved the event data in the journal and the id is 1-65534.
It means it can save 65534 records. Actually, we found the file size of single journal file is around 8MB(the default of journal file is 64MB, it will be separated by 8 files). We cannot show all SELs(65534 records) in IPMI. it might be a bug here if we don't limit the maximum.Is it necessary to limit the number of SELs to smaller?
Hi Will,
The reason that value is left at 65534 is because in the journal we don't have a way to determine how many total SEL entries are possible or how much space is left.
With the journal set at 64MB we could see more than 30000 entries. When I made the journal persistent at 6MB, we could see anywhere between 1200-2000 entries.
We could choose a smaller number for IPMI to report, but it will be arbitrary because the journal size can be changed without notifying IPMI.
However, because of the size (1200 entries in 6MB) and other limitations with a persistent journal, I'm experimenting with using rsyslog to log SEL entries into a syslog file. This will allow the journal to remain volatile while persisting only the logs that we need to. It will also have a more consistent size, so it will be possible to fix the size to a smaller number more accurately as you suggest.
You can see my current rsyslog changes here: https://gerrit.openbmc-project.xyz/q/topic:"rsyslog+SEL"+(status:open%20OR%20status:merged)
If you are interested, I will also share my rsyslog configuration, but I was planning to test it out internally for a while to make sure it will work so I don't cause thrash if it has to change again.
Thanks,
-Jason
Quoted 4 lines
BRs,
Will
Quoted 34 lines · Will Liang (梁永鉉)
Quoted 3 lines
Hi,
Our BMC is using package "phosphor-sel-logger" to support IPMI SEL. it
saved the event data in the journal and the id is 1-65534.
Quoted 1 line
It means it can save 65534 records. Actually, we found the file size of single
journal file is around 8MB(the default of journal file is 64MB, it will be separated by 8 files).
Quoted 1 line
We cannot show all SELs(65534 records) in IPMI. it might be a bug here if we
don't limit the maximum.
Quoted 2 lines
Is it necessary to limit the number of SELs to smaller?
Hi Will,
The reason that value is left at 65534 is because in the journal we don't have a way to determine how many total SEL entries are possible or how much space is left.
With the journal set at 64MB we could see more than 30000 entries. When I made the journal persistent at 6MB, we could see anywhere between 1200-2000 entries.
We could choose a smaller number for IPMI to report, but it will be arbitrary because the journal size can be changed without notifying IPMI.
However, because of the size (1200 entries in 6MB) and other limitations with a persistent journal, I'm experimenting with using rsyslog to log SEL entries into a syslog file. This will allow the journal to remain volatile while persisting only the logs that we need to. It will also have a more consistent size, so it will be possible to fix the size to a smaller number more accurately as you suggest.
Quoted 7 lines
You can see my current rsyslog changes here:
https://gerrit.openbmc-project.xyz/q/topic:"rsyslog+SEL"+(status:open%20OR%
20status:merged)If you are interested, I will also share my rsyslog configuration, but I was planning to test it out internally for a while to make sure it will work so I don't cause thrash if it has to change again.
Hi Jason,
I am very thankful that you reply me quickly and it's great to hear that you are working on this issue. If you can share your rsyslog configuration with me that will be a good news for me, I can trace it first.
According to the above URL, is it something I missed, I did not find the "add sel" to sys log function? I wander to know how you add sel to sys log. Or you want to share it after internal testing.
Thanks,
Will
Quoted 6 lines
Thanks,
-JasonQuoted 4 lines
BRs,
Will
Quoted 4 lines · Will Liang (梁永鉉)
Hi Jason,
I am very thankful that you reply me quickly and it's great to hear that you are working on this issue. If you can share your rsyslog configuration with me that will be a good news for me, I can trace it first.
I will share my rsyslog configuration file below. It requires the imjournal module to be installed which can be done by adding 'PACKAGECONFIG_append = " imjournal"' to an rsyslog_%.bbappend file.
Quoted 2 lines
According to the above URL, is it something I missed, I did not find the "add sel" to sys log function?
If the imjournal rsyslog module works well enough, then the "add sel" function won't need to change. SEL entries will still be added to the journal and the imjournal module will read the journal to find new SEL entries for rsyslog to save to the SEL log file.
Quoted 2 lines
I wander to know how you add sel to sys log. Or you want to share it after internal testing.
I'm willing to share whatever you are interested in, but it may change if internal testing finds issues. I'm planning to push the entire solution once we complete initial internal testing.
For clearing the log to work, you need to add "Reload" support to the rsyslog service file. I do this by adding this line to an override .conf file:
ExecReload=/bin/kill -HUP $MAINPID
Here is my rsyslog.conf file which configures the entries the imjournal module is looking for and how to format them in the syslog files:
# if you experience problems, check # http://www.rsyslog.com/troubleshoot for assistance
# rsyslog v3: load input modules # If you do not load inputs, nothing happens! # You may need to set the module load path if modules are not found. # # Ported from debian's sysklogd.conf
# Journal-style logging
# Limit to no more than 2000 entries in one minute and enable the
# journal workaround to avoid duplicate entries
module(load="imjournal" StateFile="/var/log/state"
RateLimit.Interval="60"
RateLimit.Burst="2000"
WorkAroundJournalBug="on")# Template for IPMI SEL messages
# "<timestamp> <ID>,<Type>,<EventData>,[<Generator ID>,<Path>,<Direction>]"
template(name="IPMISELTemplate" type="list") {
property(name="timereported" dateFormat="rfc3339")
constant(value=" ")
property(name="$!IPMI_SEL_RECORD_ID")
constant(value=",")
property(name="$!IPMI_SEL_RECORD_TYPE")
constant(value=",")
property(name="$!IPMI_SEL_DATA")
constant(value=",")
property(name="$!IPMI_SEL_GENERATOR_ID")
constant(value=",")
property(name="$!IPMI_SEL_SENSOR_PATH")
constant(value=",")
property(name="$!IPMI_SEL_EVENT_DIR")
constant(value="\n")
}# Template for Redfish messages
# "<timestamp> <MessageId>,<MessageArgs>"
template(name="RedfishTemplate" type="list") {
property(name="timereported" dateFormat="rfc3339")
constant(value=" ")
property(name="$!REDFISH_MESSAGE_ID")
constant(value=",")
property(name="$!REDFISH_MESSAGE_ARGS")
constant(value="\n")
}# If the journal entry has the IPMI SEL MESSAGE_ID, save as IPMI SEL if ($!MESSAGE_ID == "b370836ccf2f4850ac5bee185b77893a") then {
action(type="omfile" file="/var/log/ipmi_sel"
template="IPMISELTemplate")
}
# If the journal entry has a Redfish MessageId, save as a Redfish event if ($!REDFISH_MESSAGE_ID != "") then {
action(type="omfile" file="/var/log/redfish" template="RedfishTemplate")
}
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf