Skip to contentSkip to conversation

3.0 -> 3.10 regression? poweroff instead of reboot?

DISCUSSION Nikola Ciprich 2 participants 15 messages Mar 10, 2014 – Apr 8, 2014

lore
Share & tools
lore
#0 lore raw

Hi,

after installing new server, we've noticed weird behaviour: when box is about to reboot, it powers off instead. This was with 3.10.33, downgrading to 3.0.101 seems to fix this problem. I didn't have time to play with it today and box had to be shipped today, so it'll take few days before I'll have chance for further research, but I wanted to ask whether somebody has met similar problem and what I should check first?

The system is x86_64 centos 6, HW is supermicro x9sci-ln4f.

Any help appreciated

When I have chance to put my hands on this box again, I'll try latest kernel and I can also do bisect, if needed.

BR

nik

Signature · 13 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#1 lore raw

(CC Tomas Winkler)

So finally I got some time to look deeper into this, and the problem might not be regression after all.. It is caused by mei_me module, blacklisting it "fixes it" for me.

So for me it's kinda of solution, although I guess MEI related people might be interested in it..

If I could be of any help, please let me know - at least I've got hardware on which I can reproduce the problem..

BR

nik

Quoted 37 lines · Nikola Ciprich

Hi,

after installing new server, we've noticed weird behaviour: when box is about to reboot, it powers off instead. This was with 3.10.33, downgrading to 3.0.101 seems to fix this problem. I didn't have time to play with it today and box had to be shipped today, so it'll take few days before I'll have chance for further research, but I wanted to ask whether somebody has met similar problem and what I should check first?

The system is x86_64 centos 6, HW is supermicro x9sci-ln4f.

Any help appreciated

When I have chance to put my hands on this box again, I'll try latest kernel and I can also do bisect, if needed.

BR

nik

Signature · 13 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
Signature · 14 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#2 lore raw
Quoted 12 lines

(CC Tomas Winkler)

So finally I got some time to look deeper into this, and the problem might not be regression after all.. It is caused by mei_me module, blacklisting it "fixes it" for me.

So for me it's kinda of solution, although I guess MEI related people might be interested in it..

If I could be of any help, please let me know - at least I've got hardware on which I can reproduce the problem..

Quoted 25 lines

BR

nik

Quoted 19 lines · Nikola Ciprich

Hi,

after installing new server, we've noticed weird behaviour: when box is about to reboot, it powers off instead. This was with 3.10.33, downgrading to 3.0.101 seems to fix this problem. I didn't have time to play with it today and box had to be shipped today, so it'll take few days before I'll have chance for further research, but I wanted to ask whether somebody has met similar problem and what I should check first?

The system is x86_64 centos 6, HW is supermicro x9sci-ln4f.

Any help appreciated

When I have chance to put my hands on this box again, I'll try latest kernel and I can also do bisect, if needed.

BR

nik

If this is a server platform can you please try this patch https://lkml.org/lkml/2014/3/25/396

Thanks
Tomas

#3 lore raw

Hi Tomas,

Quoted 1 line

If this is a server platform can you please try this patch https://lkml.org/lkml/2014/3/25/396

tried 3.10.35 with Your patch and now I can see following in dmesg:

[ +0.000290] mei_me 0000:00:16.0: Device doesn't have valid ME Interface [ +0.007656] mei_me 0000:00:16.0: initialization failed.

now reboot works as expected. Once this hits ustream (if it hasn't already), this could go to -stable, do You agree?

thanks!

nik

Quoted 5 lines

Thanks
Tomas

Signature · 13 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#4 lore raw
Quoted 9 lines

Hi Tomas,

Quoted 1 line

If this is a server platform can you please try this patch

https://lkml.org/lkml/2014/3/25/396

tried 3.10.35 with Your patch and now I can see following in dmesg:

Quoted 8 lines

[ +0.000290] mei_me 0000:00:16.0: Device doesn't have valid ME Interface [ +0.007656] mei_me 0000:00:16.0: initialization failed.

now reboot works as expected. Once this hits ustream (if it hasn't already), this could go to -stable, do You agree?

thanks!

I definitely asked for adding it to stable. Would you wish to add you tested-by to the patch?. Tomas

#5 lore raw

Hi,

sure! count me in.

n.

Quoted 21 lines · Tomas
Quoted 9 lines

Hi Tomas,

Quoted 1 line

If this is a server platform can you please try this patch

https://lkml.org/lkml/2014/3/25/396

tried 3.10.35 with Your patch and now I can see following in dmesg:

Quoted 8 lines

[ +0.000290] mei_me 0000:00:16.0: Device doesn't have valid ME Interface [ +0.007656] mei_me 0000:00:16.0: initialization failed.

now reboot works as expected. Once this hits ustream (if it hasn't already), this could go to -stable, do You agree?

thanks!

I definitely asked for adding it to stable. Would you wish to add you tested-by to the patch?. Tomas

Signature · 13 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#6 lore raw
In reply to Winkler, Tomas

Hi Tomas,

bad news, we just had failed boot with 3.10.35 with your patch:

last line was:

mei_me 0000:00:16.0: wait hw ready failed. status = -110

so it can probably happen that boot hangs on mei_me init attempt..

colleague just told me it also happened to him with 3.13 kernel, however I don't have more info on that case..

if I can provide more information, please let me know.

BR

nik

Quoted 21 lines · Tomas
Quoted 9 lines

Hi Tomas,

Quoted 1 line

If this is a server platform can you please try this patch

https://lkml.org/lkml/2014/3/25/396

tried 3.10.35 with Your patch and now I can see following in dmesg:

Quoted 8 lines

[ +0.000290] mei_me 0000:00:16.0: Device doesn't have valid ME Interface [ +0.007656] mei_me 0000:00:16.0: initialization failed.

now reboot works as expected. Once this hits ustream (if it hasn't already), this could go to -stable, do You agree?

thanks!

I definitely asked for adding it to stable. Would you wish to add you tested-by to the patch?. Tomas

Signature · 13 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#7 lore raw
Quoted 9 lines

Hi Tomas,

bad news, we just had failed boot with 3.10.35 with your patch:

last line was:

Quoted 9 lines

mei_me 0000:00:16.0: wait hw ready failed. status = -110

so it can probably happen that boot hangs on mei_me init attempt..

colleague just told me it also happened to him with 3.13 kernel, however I don't have more info on that case..

if I can provide more information, please let me know.

I would need the devices ids of the MEI device on your platform (lspci will do the job), is this same platform as you tested before? You should not get to requesting the HW readiness w/ my patch.
Thanks
Tomas

-----------------------------

#8 lore raw

Hello Tomas,

Quoted 2 lines

I would need the devices ids of the MEI device on your platform (lspci will do the job), is this same platform as you tested before? You should not get to requesting the HW readiness w/ my patch.

yes, it's the same box.

here are the IDs:

00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04) 00:16.1 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #2 (rev 04)

00:16.0 0780: 8086:1c3a (rev 04)
00:16.1 0780: 8086:1c3b (rev 04)

nik

Quoted 5 lines

Thanks
Tomas

-----------------------------

Signature · 14 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#9 lore raw
Quoted 12 lines

Hello Tomas,

Quoted 1 line

I would need the devices ids of the MEI device on your platform (lspci will do

the job), is this same platform as you tested before?

Quoted 1 line

You should not get to requesting the HW readiness w/ my patch.

yes, it's the same box.

here are the IDs:

Quoted 7 lines

00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:16.1 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #2 (rev 04)

00:16.0 0780: 8086:1c3a (rev 04)
00:16.1 0780: 8086:1c3b (rev 04)

Are use you have the patch applied, because the driver just should not load. If any case you dump the firmware status registers"
At offset 0x40 and 0x48 in PCI config space
sudo hexdump -s 0x48 -n 4 /proc/bus/pci/00/16.0
sudo hexdump -s 0x40 -n 4 /proc/bus/pci/00/16.0

Thanks
Tomas

#10 lore raw

Hi Tomas,

yes, I just checked source rpm and it's applied..

here are the dumps:

[root@k310test ~]# hexdump -s 0x48 -n 4 /proc/bus/pci/00/16.0 0000048 a301 3900
000004c
[root@k310test ~]# hexdump -s 0x40 -n 4 /proc/bus/pci/00/16.0 0000040 0345 000f
0000044

BR

nik

Quoted 10 lines

Are use you have the patch applied, because the driver just should not load. If any case you dump the firmware status registers"
At offset 0x40 and 0x48 in PCI config space
sudo hexdump -s 0x48 -n 4 /proc/bus/pci/00/16.0
sudo hexdump -s 0x40 -n 4 /proc/bus/pci/00/16.0

Thanks
Tomas

Signature · 13 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#11 lore raw
Quoted 14 lines

-----Original Message-----
From: Nikola Ciprich [mailto:[email protected]]
Sent: Monday, April 07, 2014 16:26
To: Winkler, Tomas
Cc: Greg KH ([email protected]); [email protected]; Nikola Ciprich
Subject: Re: 3.0 -> 3.10 regression? poweroff instead of reboot?

Hi Tomas,

yes, I just checked source rpm and it's applied..

Quoted 8 lines

here are the dumps:

[root@k310test ~]# hexdump -s 0x48 -n 4 /proc/bus/pci/00/16.0 0000048 a301 3900
000004c
[root@k310test ~]# hexdump -s 0x40 -n 4 /proc/bus/pci/00/16.0 0000040 0345 000f
0000044

I'm not sure how this is possible that this is failing for you

if ((0xa301 & 0x600) == 0x200)

	goto no_mei; 

so this is okay, the driver should not bind

How the patch worked for you before this test?

Sorry for asking but can you please double check your compilation setup.

Thanks
Tomas

#12 lore raw

Hello Tomas,

Quoted 9 lines

I'm not sure how this is possible that this is failing for you

if ((0xa301 & 0x600) == 0x200)

	goto no_mei; 

so this is okay, the driver should not bind

How the patch worked for you before this test?

Sorry for asking but can you please double check your compilation setup.

You're right of course :-(

I tried rebuilding rpm again now, and patch spits following:

Hunk #1 succeeded at 105 (offset -2 lines).
patch unexpectedly ends in middle of line

it's bad rpm build didn't abort at this point, but apparently this is patch's fault:

[root@dev6 linux-3.10]# cat /tmp/mei/1 | patch -sp1 patch unexpectedly ends in middle of line [root@dev6 linux-3.10]# echo $?
0

:-(

well, this sucks, I really wouldn't expect that kind of bug in such widely used program as patch :(

I'll report that.

(btw this error was caused by one whitespace character at last line of patch)

anyways, You can certainly forgot about that, it was my fault :(

sorry about that

nik

Quoted 4 lines

Thanks
Tomas

Signature · 13 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------
#13 lore raw
Quoted 28 lines

-----Original Message-----
From: Nikola Ciprich [mailto:[email protected]]
Sent: Tuesday, April 08, 2014 09:50
To: Winkler, Tomas
Cc: Greg KH ([email protected]); [email protected]; Nikola Ciprich
Subject: Re: 3.0 -> 3.10 regression? poweroff instead of reboot?

Quoted 1 line

Sorry for asking but can you please double check your compilation setup.

You're right of course :-(

I tried rebuilding rpm again now, and patch spits following:

Hunk #1 succeeded at 105 (offset -2 lines).
patch unexpectedly ends in middle of line

it's bad rpm build didn't abort at this point, but apparently this is patch's fault:

[root@dev6 linux-3.10]# cat /tmp/mei/1 | patch -sp1 patch unexpectedly ends in middle of line [root@dev6 linux-3.10]# echo $?
0

:-(

Quoted 7 lines

well, this sucks, I really wouldn't expect that kind of bug in such widely used program as patch :(

I'll report that.

(btw this error was caused by one whitespace character at last line of patch)

This is really strange, can it be that your mailer has introduced some blanks into the patch, I've checked that patch applies to all relevant stable kernels.

Quoted 3 lines

anyways, You can certainly forgot about that, it was my fault :(

sorry about that

No harm done, it's good this was sorted out.
Thanks
Tomas

#14 lore raw
Quoted 2 lines

This is really strange, can it be that your mailer has introduced some blanks into the patch, I've checked that patch applies to all relevant stable kernels.

I had it downloaded from link You've sent:
https://lkml.org/lkml/2014/3/25/396

using

wget https://lkml.org/lkml/diff/2014/3/25/396/1

I get two whitespaces on the last line..

n.

Quoted 11 lines
Quoted 4 lines

anyways, You can certainly forgot about that, it was my fault :(

sorry about that

No harm done, it's good this was sorted out.
Thanks
Tomas

Signature · 13 lines
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28.rijna 168, 709 00 Ostrava

tel.:   +420 591 166 214
fax:    +420 596 621 273
mobil:  +420 777 093 799
www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: [email protected]
-------------------------------------