3.0 -> 3.10 regression? poweroff instead of reboot?
Hi,
after installing new server, we've noticed weird behaviour: when box is about to reboot, it powers off instead. This was with 3.10.33, downgrading to 3.0.101 seems to fix this problem. I didn't have time to play with it today and box had to be shipped today, so it'll take few days before I'll have chance for further research, but I wanted to ask whether somebody has met similar problem and what I should check first?
The system is x86_64 centos 6, HW is supermicro x9sci-ln4f.
Any help appreciated
When I have chance to put my hands on this box again, I'll try latest kernel and I can also do bisect, if needed.
BR
nik
Signature · 13 lines
------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
(CC Tomas Winkler)
So finally I got some time to look deeper into this, and the problem might not be regression after all.. It is caused by mei_me module, blacklisting it "fixes it" for me.
So for me it's kinda of solution, although I guess MEI related people might be interested in it..
If I could be of any help, please let me know - at least I've got hardware on which I can reproduce the problem..
BR
nik
Quoted 37 lines · Nikola Ciprich
Hi,
after installing new server, we've noticed weird behaviour: when box is about to reboot, it powers off instead. This was with 3.10.33, downgrading to 3.0.101 seems to fix this problem. I didn't have time to play with it today and box had to be shipped today, so it'll take few days before I'll have chance for further research, but I wanted to ask whether somebody has met similar problem and what I should check first?
The system is
x86_64centos 6, HW is supermicro x9sci-ln4f.Any help appreciated
When I have chance to put my hands on this box again, I'll try latest kernel and I can also do bisect, if needed.
BR
nik
Signature · 13 lines
------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Signature · 14 lines
------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Quoted 12 lines
(CC Tomas Winkler)
So finally I got some time to look deeper into this, and the problem might not be regression after all.. It is caused by
mei_memodule, blacklisting it "fixes it" for me.So for me it's kinda of solution, although I guess MEI related people might be interested in it..
If I could be of any help, please let me know - at least I've got hardware on which I can reproduce the problem..
Quoted 25 lines
BR
nik
Quoted 19 lines · Nikola Ciprich
Hi,
after installing new server, we've noticed weird behaviour: when box is about to reboot, it powers off instead. This was with 3.10.33, downgrading to 3.0.101 seems to fix this problem. I didn't have time to play with it today and box had to be shipped today, so it'll take few days before I'll have chance for further research, but I wanted to ask whether somebody has met similar problem and what I should check first?
The system is
x86_64centos 6, HW is supermicro x9sci-ln4f.Any help appreciated
When I have chance to put my hands on this box again, I'll try latest kernel and I can also do bisect, if needed.
BR
nik
If this is a server platform can you please try this patch https://lkml.org/lkml/2014/3/25/396
Thanks
Tomas
Hi Tomas,
Quoted 1 line
If this is a server platform can you please try this patch https://lkml.org/lkml/2014/3/25/396
tried 3.10.35 with Your patch and now I can see following in dmesg:
[ +0.000290] mei_me 0000:00:16.0: Device doesn't have valid ME Interface [ +0.007656] mei_me 0000:00:16.0: initialization failed.
now reboot works as expected. Once this hits ustream (if it hasn't already), this could go to -stable, do You agree?
thanks!
nik
Quoted 5 lines
Thanks
Tomas
Signature · 13 lines
------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Quoted 9 lines
Hi Tomas,
Quoted 1 line
If this is a server platform can you please try this patch
https://lkml.org/lkml/2014/3/25/396
tried 3.10.35 with Your patch and now I can see following in dmesg:
Quoted 8 lines
[ +0.000290]
mei_me0000:00:16.0: Device doesn't have valid ME Interface [ +0.007656]mei_me0000:00:16.0: initialization failed.now reboot works as expected. Once this hits ustream (if it hasn't already), this could go to -stable, do You agree?
thanks!
I definitely asked for adding it to stable. Would you wish to add you tested-by to the patch?. Tomas
Hi,
sure! count me in.
n.
Quoted 21 lines · Tomas
Quoted 9 lines
Hi Tomas,
Quoted 1 line
If this is a server platform can you please try this patch
https://lkml.org/lkml/2014/3/25/396
tried 3.10.35 with Your patch and now I can see following in dmesg:
Quoted 8 lines
[ +0.000290]
mei_me0000:00:16.0: Device doesn't have valid ME Interface [ +0.007656]mei_me0000:00:16.0: initialization failed.now reboot works as expected. Once this hits ustream (if it hasn't already), this could go to -stable, do You agree?
thanks!
I definitely asked for adding it to stable. Would you wish to add you tested-by to the patch?. Tomas
Signature · 13 lines
------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Hi Tomas,
bad news, we just had failed boot with 3.10.35 with your patch:
last line was:
mei_me 0000:00:16.0: wait hw ready failed. status = -110
so it can probably happen that boot hangs on mei_me init attempt..
colleague just told me it also happened to him with 3.13 kernel, however I don't have more info on that case..
if I can provide more information, please let me know.
BR
nik
Quoted 21 lines · Tomas
Quoted 9 lines
Hi Tomas,
Quoted 1 line
If this is a server platform can you please try this patch
https://lkml.org/lkml/2014/3/25/396
tried 3.10.35 with Your patch and now I can see following in dmesg:
Quoted 8 lines
[ +0.000290]
mei_me0000:00:16.0: Device doesn't have valid ME Interface [ +0.007656]mei_me0000:00:16.0: initialization failed.now reboot works as expected. Once this hits ustream (if it hasn't already), this could go to -stable, do You agree?
thanks!
I definitely asked for adding it to stable. Would you wish to add you tested-by to the patch?. Tomas
Signature · 13 lines
------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Quoted 9 lines
Hi Tomas,
bad news, we just had failed boot with 3.10.35 with your patch:
last line was:
Quoted 9 lines
mei_me0000:00:16.0: wait hw ready failed. status = -110so it can probably happen that boot hangs on
mei_meinit attempt..colleague just told me it also happened to him with 3.13 kernel, however I don't have more info on that case..
if I can provide more information, please let me know.
I would need the devices ids of the MEI device on your platform (lspci will do the job), is this same platform as you tested before? You should not get to requesting the HW readiness w/ my patch.
Thanks
Tomas
-----------------------------
Hello Tomas,
Quoted 2 lines
I would need the devices ids of the MEI device on your platform (lspci will do the job), is this same platform as you tested before? You should not get to requesting the HW readiness w/ my patch.
yes, it's the same box.
here are the IDs:
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04) 00:16.1 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #2 (rev 04)
00:16.0 0780: 8086:1c3a (rev 04)
00:16.1 0780: 8086:1c3b (rev 04)
nik
Quoted 5 lines
Thanks
Tomas-----------------------------
Signature · 14 lines
------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Quoted 12 lines
Hello Tomas,
Quoted 1 line
I would need the devices ids of the MEI device on your platform (lspci will do
the job), is this same platform as you tested before?
Quoted 1 line
You should not get to requesting the HW readiness w/ my patch.
yes, it's the same box.
here are the IDs:
Quoted 7 lines
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #1 (rev 04)
00:16.1 Communication controller: Intel Corporation 6 Series/C200 Series Chipset Family MEI Controller #2 (rev 04)00:16.0 0780: 8086:1c3a (rev 04)
00:16.1 0780: 8086:1c3b (rev 04)
Are use you have the patch applied, because the driver just should not load. If any case you dump the firmware status registers"
At offset 0x40 and 0x48 in PCI config space
sudo hexdump -s 0x48 -n 4 /proc/bus/pci/00/16.0
sudo hexdump -s 0x40 -n 4 /proc/bus/pci/00/16.0
Thanks
Tomas
Hi Tomas,
yes, I just checked source rpm and it's applied..
here are the dumps:
[root@k310test ~]# hexdump -s 0x48 -n 4 /proc/bus/pci/00/16.0 0000048 a301 3900
000004c
[root@k310test ~]# hexdump -s 0x40 -n 4 /proc/bus/pci/00/16.0 0000040 0345 000f
0000044
BR
nik
Quoted 10 lines
Are use you have the patch applied, because the driver just should not load. If any case you dump the firmware status registers"
At offset0x40and0x48in PCI config space
sudo hexdump -s0x48-n 4/proc/bus/pci/00/16.0
sudo hexdump -s0x40-n 4/proc/bus/pci/00/16.0Thanks
Tomas
Signature · 13 lines
------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Quoted 14 lines
-----Original Message-----
From: Nikola Ciprich [mailto:[email protected]]
Sent: Monday, April 07, 2014 16:26
To: Winkler, Tomas
Cc: Greg KH ([email protected]); [email protected]; Nikola Ciprich
Subject: Re: 3.0 -> 3.10 regression? poweroff instead of reboot?Hi Tomas,
yes, I just checked source rpm and it's applied..
Quoted 8 lines
here are the dumps:
[root@k310test ~]# hexdump -s
0x48-n 4/proc/bus/pci/00/16.00000048 a301 3900
000004c
[root@k310test ~]# hexdump -s0x40-n 4/proc/bus/pci/00/16.00000040 0345 000f
0000044
I'm not sure how this is possible that this is failing for you
if ((0xa301 & 0x600) == 0x200)
goto no_mei;
so this is okay, the driver should not bind
How the patch worked for you before this test?
Sorry for asking but can you please double check your compilation setup.
Thanks
Tomas
Hello Tomas,
Quoted 9 lines
I'm not sure how this is possible that this is failing for you
if ((
0xa301&0x600) ==0x200)goto no_mei;so this is okay, the driver should not bind
How the patch worked for you before this test?
Sorry for asking but can you please double check your compilation setup.
You're right of course :-(
I tried rebuilding rpm again now, and patch spits following:
Hunk #1 succeeded at 105 (offset -2 lines).
patch unexpectedly ends in middle of line
it's bad rpm build didn't abort at this point, but apparently this is patch's fault:
[root@dev6 linux-3.10]# cat /tmp/mei/1 | patch -sp1 patch unexpectedly ends in middle of line [root@dev6 linux-3.10]# echo $?
0
:-(
well, this sucks, I really wouldn't expect that kind of bug in such widely used program as patch :(
I'll report that.
(btw this error was caused by one whitespace character at last line of patch)
anyways, You can certainly forgot about that, it was my fault :(
sorry about that
nik
Quoted 4 lines
Thanks
Tomas
Signature · 13 lines
------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------
Quoted 28 lines
-----Original Message-----
From: Nikola Ciprich [mailto:[email protected]]
Sent: Tuesday, April 08, 2014 09:50
To: Winkler, Tomas
Cc: Greg KH ([email protected]); [email protected]; Nikola Ciprich
Subject: Re: 3.0 -> 3.10 regression? poweroff instead of reboot?Quoted 1 line
Sorry for asking but can you please double check your compilation setup.
You're right of course :-(
I tried rebuilding rpm again now, and patch spits following:
Hunk #1 succeeded at 105 (offset -2 lines).
patch unexpectedly ends in middle of lineit's bad rpm build didn't abort at this point, but apparently this is patch's fault:
[root@dev6 linux-3.10]# cat
/tmp/mei/1| patch -sp1 patch unexpectedly ends in middle of line [root@dev6 linux-3.10]# echo $?
0:-(
Quoted 7 lines
well, this sucks, I really wouldn't expect that kind of bug in such widely used program as patch :(
I'll report that.
(btw this error was caused by one whitespace character at last line of patch)
This is really strange, can it be that your mailer has introduced some blanks into the patch, I've checked that patch applies to all relevant stable kernels.
Quoted 3 lines
anyways, You can certainly forgot about that, it was my fault :(
sorry about that
No harm done, it's good this was sorted out.
Thanks
Tomas
Quoted 2 lines
This is really strange, can it be that your mailer has introduced some blanks into the patch, I've checked that patch applies to all relevant stable kernels.
I had it downloaded from link You've sent:
https://lkml.org/lkml/2014/3/25/396
using
wget https://lkml.org/lkml/diff/2014/3/25/396/1
I get two whitespaces on the last line..
n.
Quoted 11 lines
Quoted 4 lines
anyways, You can certainly forgot about that, it was my fault :(
sorry about that
No harm done, it's good this was sorted out.
Thanks
Tomas
Signature · 13 lines
------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: [email protected] -------------------------------------