Skip to contentSkip to conversation

Why can't /dev/ipmi0 on host be created stably

DISCUSSION xiuzhi 2 participants 3 messages Jan 17, 2019 – Jan 21, 2019

lore
Share & tools
lore

xiuzhi

author
#0 lore raw

Hi Oskar,

Quoted 2 lines

Hi>
I'm not sure what CA3 is, I'm afraid. Can you explain? Or do you have a reference?

The CA3 is the SM BIOS Base Address of ipmi-kcs /dev/kcs3, please see the Table C1-3, Byte-aligned I/O Mapped Register Address examples in Appendix C1 - Locating IPMI System Interfaces via SM BIOS Tables of Intelligent Platform Management Interface Specification V2.0.

Quoted 4 lines

AMI BIOS can be configured to try a number of times to talk with the >BMC. To check whether it's just a timing issue, reboot the host once the >BMC is up and running.

Is that the case for you, too?

I will try to config the AMI BIOS and test again.
In my case, the CA3 value would not change from 0x04 to 0x00 until shut down the power and wait more than 40 seconds ,then reboot .

Quoted 1 line

Also, I'm not sure whether the problem I observed is even applicable to

KCS.

Quoted 1 line

Oskar.

Best,
Xiuzhi

On Wed, Jan 16, 2019, 9:27 PM xiuzhi <[email protected] wrote:

Hi Oskar,
I asked the BIOS developer, he said the BIOS read the io CA3 firstly, if it is not 0x00, the BIOS will think the ipmi doesn't exist.
I input the command " mm -io CA3" in BIOS, it return the 0x04 every times, so it's pointless for BIOS to wait the BMC answer. I user ipmi-kcs mode.

Best,
xiuzhi
Hi Oskar,
I use opensource UEFI BIOS and AMI BIOS to test. I am able to modify the code of UEFI BIOS.
How should I change the BIOS code to ensure the synchronization between the two sides so that I can tell my BIOS team to modify it?

Best,
xiuzhi

------------------ Original ------------------
From: "Oskar Senft";<[email protected]>;
Send time: Wednesday, Jan 16, 2019 10:10 PM
To: "xiuzhi"<[email protected]>;
Cc: "openbmc"<[email protected]>;
Subject: Re: Why can't /dev/ipmi0 on host be created stably

Hi Xiuzhi

The following observation may or may not be related to your problem:

We're using AMI BIOS with OpenBMC using the IPMI BT mode. I found that in certain cases the host and BMC can get "out of sync" wrt. what the BMC put in the message queue and what the BIOS expects. The problem can be caused by the BMC responding "slowly" to an IPMI request from the BIOS where the BIOS times out before the BMC responds. The BMC then responds before the BIOS issues the next request, which causes the BIOS to read the previous response as the response for the latest request.

This has resulted in all sorts of weird behavior for us, incl. the BIOS not "detecting" the BMC and therefore not exposing it in the SMBIOS tables, which - in our case - is the configuration source for the Linux IPMI driver.

I hope this helps?

What BIOS are you using? Are you able to make changes to the BIOS code?

Oskar.

On Tue, Jan 15, 2019 at 9:33 PM xiuzhi <[email protected]> wrote:

Hi All,
I am testing the bmc ipmi-kcs reliability.
The /dev/ipmi0 on host will not be available in several consecutive bmc reboot test. The CA3 value is 0x04 which can be got by the command "mm -io ca3" on host bios shell when failing to create the ipmi0 .
The CA3 value 0x04 means that SMS_ATN is set to 1, which meas:
"Set to 1 when the BMC has one or more messages in the Receive Message Queue, or when a watchdog timer pre-timeout, or event message buffer full condition exists [2] . OEMs may also elect to set this flag is one of the OEM 1, 2,or 3 flags from the Get Message Flags command becomes set. R/O This bit is related to indicating when the BMC is the source of a system interrupt. Refer to sections 9.12, KCS Communication and Non-communication , 9.13, Physical Interrupt Line Sharing, and 9.14, Additional Specifications for the KCS interface for additional information on the use and requirements for the SMS_ATN bit."

see the Table 9-, KCS Interface Status Register Bits in section 9.7 of Intelligent Platform ManageBment Interface Specification .
What causes this problem?
Best,
Xiuzhi

#1 lore raw

Hi Xiuzhi,
Check if you have any GPIO which needs configuration like BMC READY which tells BIOS that BMC is ready to communicate. Run kcs with –vv option and see if you are getting any communication with Bios.

I also use AMI Bios and we have this GPIO between BIOS and BMC.

Regards
-Vijay

From: openbmc <[email protected]> on behalf of xiuzhi <[email protected]>
Date: Wednesday, January 16, 2019 at 7:32 PM
To: Oskar Senft <[email protected]>
Cc: openbmc <[email protected]>
Subject: Re: Re: Why can't /dev/ipmi0 on host be created stably

Hi Oskar,

Quoted 3 lines

Hi

I'm not sure what CA3 is, I'm afraid. Can you explain? Or do you have a reference?

The CA3 is the SM BIOS Base Address of ipmi-kcs /dev/kcs3, please see the Table C1-3, Byte-aligned I/O Mapped Register Address examples in Appendix C1 - Locating IPMI System Interfaces via SM BIOS Tables of Intelligent Platform Management Interface Specification V2.0.

Quoted 3 lines

AMI BIOS can be configured to try a number of times to talk with the >BMC. To check whether it's just a timing issue, reboot the host once the >BMC is up and running.

Is that the case for you, too?

I will try to config the AMI BIOS and test again.
In my case, the CA3 value would not change from 0x04 to 0x00 until shut down the power and wait more than 40 seconds ,then reboot .

Quoted 1 line

Also, I'm not sure whether the problem I observed is even applicable to

KCS.

Quoted 1 line

Oskar.

Best,
Xiuzhi

On Wed, Jan 16, 2019, 9:27 PM xiuzhi <[email protected]<mailto:[email protected]> wrote:
Hi Oskar,
I asked the BIOS developer, he said the BIOS read the io CA3 firstly, if it is not 0x00, the BIOS will think the ipmi doesn't exist.
I input the command " mm -io CA3" in BIOS, it return the 0x04 every times, so it's pointless for BIOS to wait the BMC answer. I user ipmi-kcs mode.

Best,
xiuzhi
Hi Oskar,
I use opensource UEFI BIOS and AMI BIOS to test. I am able to modify the code of UEFI BIOS.
How should I change the BIOS code to ensure the synchronization between the two sides so that I can tell my BIOS team to modify it?

Best,
xiuzhi

------------------ Original ------------------
From: "Oskar Senft";<[email protected]<mailto:[email protected]>>; Send time: Wednesday, Jan 16, 2019 10:10 PM
To: "xiuzhi"<[email protected]<mailto:[email protected]>>;
Cc: "openbmc"<[email protected]<mailto:[email protected]>>;
Subject: Re: Why can't /dev/ipmi0 on host be created stably

Hi Xiuzhi

The following observation may or may not be related to your problem:

We're using AMI BIOS with OpenBMC using the IPMI BT mode. I found that in certain cases the host and BMC can get "out of sync" wrt. what the BMC put in the message queue and what the BIOS expects. The problem can be caused by the BMC responding "slowly" to an IPMI request from the BIOS where the BIOS times out before the BMC responds. The BMC then responds before the BIOS issues the next request, which causes the BIOS to read the previous response as the response for the latest request.

This has resulted in all sorts of weird behavior for us, incl. the BIOS not "detecting" the BMC and therefore not exposing it in the SMBIOS tables, which - in our case - is the configuration source for the Linux IPMI driver.

I hope this helps?

What BIOS are you using? Are you able to make changes to the BIOS code?

Oskar.

On Tue, Jan 15, 2019 at 9:33 PM xiuzhi <[email protected]<mailto:[email protected]>> wrote:
Hi All,
I am testing the bmc ipmi-kcs reliability.
The /dev/ipmi0 on host will not be available in several consecutive bmc reboot test. The CA3 value is 0x04 which can be got by the command "mm -io ca3" on host bios shell when failing to create the ipmi0 .
The CA3 value 0x04 means that SMS_ATN is set to 1, which meas:
"Set to 1 when the BMC has one or more messages in the Receive Message Queue, or when a watchdog timer pre-timeout, or event message buffer full condition exists [2] . OEMs may also elect to set this flag is one of the OEM 1, 2,or 3 flags from the Get Message Flags command becomes set. R/O This bit is related to indicating when the BMC is the source of a system interrupt. Refer to sections 9.12, KCS Communication and Non-communication , 9.13, Physical Interrupt Line Sharing, and 9.14, Additional Specifications for the KCS interface for additional information on the use and requirements for the SMS_ATN bit."
see the Table 9-, KCS Interface Status Register Bits in section 9.7 of Intelligent Platform ManageBment Interface Specification .
What causes this problem?
Best,
Xiuzhi

author
#2 lore raw

Hi Vijay,

Quoted 5 lines

Hi Xiuzhi,

Check if you have any GPIO which needs configuration like BMC READY which tells BIOS that BMC is ready to communicate. Run kcs with –vv option and see if you are getting any communication with Bios.

Yes, My hardware has this GPIO.

The issue is not caused by the out-sync between BMC and BIOS, because the CA3 IO would keep the 0x04 no matter how long you wait.

The ipmid should clear all of the ipmi-kcs requests from system apps when booting.

Quoted 1 line

I also use AMI Bios and we have this GPIO between BIOS and BMC.

Did your BIOS communicate to BMC successful every time in dozens of cold reboot tests?

Best,

Xiuzhi

Regards

-Vijay

From: openbmc <[email protected]> on behalf of xiuzhi <[email protected]>
Date: Wednesday, January 16, 2019 at 7:32 PM
To: Oskar Senft <[email protected]>
Cc: openbmc <[email protected]>
Subject: Re: Re: Why can't /dev/ipmi0 on host be created stably

Hi Oskar,

Quoted 6 lines

Hi

I'm not sure what CA3 is, I'm afraid. Can you explain? Or do you have a reference?

The CA3 is the SM BIOS Base Address of ipmi-kcs /dev/kcs3, please see the Table C1-3, Byte-aligned I/O Mapped Register Address examples in Appendix C1 - Locating IPMI System Interfaces via SM BIOS Tables of Intelligent Platform Management Interface Specification V2.0.

Quoted 7 lines

AMI BIOS can be configured to try a number of times to talk with the >BMC. To check whether it's just a timing issue, reboot the host once the >BMC is up and running.

Is that the case for you, too?

I will try to config the AMI BIOS and test again.

In my case, the CA3 value would not change from 0x04 to 0x00 until shut down the power and wait more than 40 seconds ,then reboot .

Quoted 1 line

Also, I'm not sure whether the problem I observed is even applicable to

KCS.

Quoted 1 line

Oskar.

Best,

Xiuzhi

On Wed, Jan 16, 2019, 9:27 PM xiuzhi <[email protected] wrote:

Hi Oskar,

I asked the BIOS developer, he said the BIOS read the io CA3 firstly, if it is not 0x00, the BIOS will think the ipmi doesn't exist.

I input the command " mm -io CA3" in BIOS, it return the 0x04 every times, so it's pointless for BIOS to wait the BMC answer.

I user ipmi-kcs mode.

Best,

xiuzhi

Hi Oskar,

I use opensource UEFI BIOS and AMI BIOS to test. I am able to modify the code of UEFI BIOS.

How should I change the BIOS code to ensure the synchronization between the two sides so that I can tell my BIOS team to modify it?

Best,

xiuzhi

------------------ Original ------------------

From: "Oskar Senft";<[email protected]>;

Send time: Wednesday, Jan 16, 2019 10:10 PM

To: "xiuzhi"<[email protected]>;

Cc "openbmc"<[email protected]>;

Subject: Re: Why can't /dev/ipmi0 on host be created stably

Hi Xiuzhi

The following observation may or may not be related to your problem:

We're using AMI BIOS with OpenBMC using the IPMI BT mode. I found that in certain cases the host and BMC can get "out of sync" wrt. what the BMC put in the message queue and what the BIOS expects. The problem can be caused by the BMC responding "slowly" to an IPMI request from the BIOS where the BIOS times out before the BMC responds. The BMC then responds before the BIOS issues the next request, which causes the BIOS to read the previous response as the response for the latest request.

This has resulted in all sorts of weird behavior for us, incl. the BIOS not "detecting" the BMC and therefore not exposing it in the SMBIOS tables, which - in our case - is the configuration source for the Linux IPMI driver.

I hope this helps?

What BIOS are you using? Are you able to make changes to the BIOS code?

Oskar.

On Tue, Jan 15, 2019 at 9:33 PM xiuzhi <[email protected]> wrote:

Hi All,

I am testing the bmc ipmi-kcs reliability.

The /dev/ipmi0 on host will not be available in several consecutive bmc reboot test. The CA3 value is 0x04 which can be got by the command "mm -io ca3" on host bios shell when failing to create the ipmi0 .

The CA3 value 0x04 means that SMS_ATN is set to 1, which meas:

"Set to 1 when the BMC has one or more messages in the Receive Message Queue, or when a watchdog timer pre-timeout, or event message buffer full condition exists [2] . OEMs may also elect to set this flag is one of the OEM 1, 2,or 3 flags from the Get Message Flags command becomes set. R/O This bit is related to indicating when the BMC is the source of a system interrupt. Refer to sections 9.12, KCS Communication and Non-communication , 9.13, Physical Interrupt Line Sharing, and 9.14, Additional Specifications for the KCS interface for additional information on the use and requirements for the SMS_ATN bit."

see the Table 9-, KCS Interface Status Register Bits in section 9.7 of Intelligent Platform ManageBment Interface Specification .

What causes this problem?

Best,

Xiuzhi