New: PCI bridge to bus boot hang at enumeration
Subsystems (3)
[+cc linux-pci, linux-kernel]
Quoted 6 lines · On Wed, Dec 28, 2022 at 08:37:52AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216859
Summary: PCI bridge to bus boot hang at enumeration
Kernel Version: 6.1-rc1
...
Quoted 6 lines
With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Same problem still exists with v6.1. and v6.2.-rc1
Thank you very much for your report, Zeno!
v6.0 works, v6.1-rc1 fails. Would you mind booting v6.1-rc1 with the "ignore_loglevel initcall_debug" kernel parameters and taking a photo when it hangs?
How did you conclude that the hang is related to a PCI bridge? I see recent PCI messages in the photo, but it looks like the last message is from NFS, so I'm wondering if I'm missing some context. The v6.0 dmesg shows several other ntfs, fuse, JFS, etc messages before more PCI-related things. Anyway, the "initcall_debug" might help us narrow it down a bit.
Bjorn
Dear Bjorn
Quoted 15 lines · Bjorn Helgaas
[+cc linux-pci, linux-kernel]
Quoted 11 lines · On Wed, Dec 28, 2022 at 08:37:52AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216859
Summary: PCI bridge to bus boot hang at enumeration
Kernel Version: 6.1-rc1
...With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Quoted 7 lines
Quoted 1 line
Same problem still exists with v6.1. and v6.2.-rc1
Thank you very much for your report, Zeno!
v6.0 works, v6.1-rc1 fails. Would you mind booting v6.1-rc1 with the "
ignore_loglevelinitcall_debug" kernel parameters and taking a photo when it hangs?
I will try this after Januar 7th 2023.
Quoted 6 lines
How did you conclude that the hang is related to a PCI bridge? I see recent PCI messages in the photo, but it looks like the last message is from NFS, so I'm wondering if I'm missing some context. The v6.0 dmesg shows several other ntfs, fuse, JFS, etc messages before more PCI-related things. Anyway, the "
initcall_debug" might help us narrow it down a bit.
I did not really conclude that. I just saw "PCI" as one of the last messages being outputted before the boot process stopped.
Best
Zeno
Quoted 20 lines · Zeno Davatz
Dear Bjorn
Quoted 17 lines · Bjorn Helgaas
[+cc linux-pci, linux-kernel]
Quoted 12 lines · On Wed, Dec 28, 2022 at 08:37:52AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216859
Summary: PCI bridge to bus boot hang at enumeration
Kernel Version: 6.1-rc1
...With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Same problem still exists with v6.1. and v6.2.-rc1
Quoted 7 lines
Quoted 5 lines
Thank you very much for your report, Zeno!
v6.0 works, v6.1-rc1 fails. Would you mind booting v6.1-rc1 with the "
ignore_loglevelinitcall_debug" kernel parameters and taking a photo when it hangs?I will try this after Januar 7th 2023.
Sounds good, thanks!
Quoted 9 lines
Quoted 6 lines
How did you conclude that the hang is related to a PCI bridge? I see recent PCI messages in the photo, but it looks like the last message is from NFS, so I'm wondering if I'm missing some context. The v6.0 dmesg shows several other ntfs, fuse, JFS, etc messages before more PCI-related things. Anyway, the "
initcall_debug" might help us narrow it down a bit.I did not really conclude that. I just saw "PCI" as one of the last messages being outputted before the boot process stopped.
OK. We'll figure it out!
Bjorn
[+cc Bruno, to include you here as well as the bugzilla]
Quoted 39 lines · Bjorn Helgaas
Quoted 15 lines · Zeno Davatz
Quoted 14 lines · Bjorn Helgaas
Quoted 12 lines · On Wed, Dec 28, 2022 at 08:37:52AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216859
Summary: PCI bridge to bus boot hang at enumeration
Kernel Version: 6.1-rc1
...With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Same problem still exists with v6.1. and v6.2.-rc1
Quoted 7 lines
Quoted 5 lines
Thank you very much for your report, Zeno!
v6.0 works, v6.1-rc1 fails. Would you mind booting v6.1-rc1 with the "
ignore_loglevelinitcall_debug" kernel parameters and taking a photo when it hangs?I will try this after Januar 7th 2023.
Sounds good, thanks!
Quoted 9 lines
Quoted 6 lines
How did you conclude that the hang is related to a PCI bridge? I see recent PCI messages in the photo, but it looks like the last message is from NFS, so I'm wondering if I'm missing some context. The v6.0 dmesg shows several other ntfs, fuse, JFS, etc messages before more PCI-related things. Anyway, the "
initcall_debug" might help us narrow it down a bit.I did not really conclude that. I just saw "PCI" as one of the last messages being outputted before the boot process stopped.
OK. We'll figure it out!
Bjorn
Dear Bjorn
Happy New Year!
Quoted 27 lines · Bjorn Helgaas
[+cc Bruno, to include you here as well as the bugzilla]
Quoted 23 lines · Bjorn Helgaas
Quoted 22 lines · Zeno Davatz
Quoted 12 lines · Bjorn Helgaas
Quoted 11 lines · On Wed, Dec 28, 2022 at 08:37:52AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216859
Summary: PCI bridge to bus boot hang at enumeration
Kernel Version: 6.1-rc1
...With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Quoted 7 lines
Quoted 1 line
Same problem still exists with v6.1. and v6.2.-rc1
Thank you very much for your report, Zeno!
v6.0 works, v6.1-rc1 fails. Would you mind booting v6.1-rc1 with the "
ignore_loglevelinitcall_debug" kernel parameters and taking a photo when it hangs?I will try this after Januar 7th 2023.
I updated the issue:
https://bugzilla.kernel.org/show_bug.cgi?id=216859
I booted with the option: "ignore_loglevel initcall_debug"
Best
Zeno
[+cc sound folks]
Quoted 20 lines · Bjorn Helgaas
Quoted 6 lines · On Wed, Dec 28, 2022 at 08:37:52AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216859
Summary: PCI bridge to bus boot hang at enumeration
Kernel Version: 6.1-rc1
...Quoted 6 lines
With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Same problem still exists with v6.1. and v6.2.-rc1
Thank you very much for your report, Zeno!
v6.0 works, v6.1-rc1 fails. Would you mind booting v6.1-rc1 with the "
ignore_loglevelinitcall_debug" kernel parameters and taking a photo when it hangs?
Quoted 6 lines
How did you conclude that the hang is related to a PCI bridge? I see recent PCI messages in the photo, but it looks like the last message is from NFS, so I'm wondering if I'm missing some context. The v6.0 dmesg shows several other ntfs, fuse, JFS, etc messages before more PCI-related things. Anyway, the "
initcall_debug" might help us narrow it down a bit.
Thanks very much for the bisection (complete log at [1])!
The bisection claims the first bad commit is:
833477fce7a1 ("Merge tag 'sound-6.1-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound")
with parents:
7e6739b9336e ("Merge tag 'drm-next-2022-10-05' of git://anongit.freedesktop.org/drm/drm")
86a4d29e7554 ("Merge tag 'asoc-v6.1' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound into for-linus")
Both 7e6739b9336e and 86a4d29e7554 tested "good" during the bisection.
There is a minor conflict when merging 86a4d29e7554 into the upstream, but I can't imagine that being resolved incorrectly.
Would you mind turning off CONFIG_SOUND in your .config and testing 833477fce7a1 again? I'm a little skeptical that the hang would be sound-related, but I guess it's a place to start.
Bjorn
Dear Bjorn
Quoted 47 lines · Bjorn Helgaas
[+cc sound folks]
Quoted 20 lines · Bjorn Helgaas
Quoted 6 lines · On Wed, Dec 28, 2022 at 08:37:52AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216859
Summary: PCI bridge to bus boot hang at enumeration
Kernel Version: 6.1-rc1
...Quoted 6 lines
With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Same problem still exists with v6.1. and v6.2.-rc1
Thank you very much for your report, Zeno!
v6.0 works, v6.1-rc1 fails. Would you mind booting v6.1-rc1 with the "
ignore_loglevelinitcall_debug" kernel parameters and taking a photo when it hangs?Quoted 6 lines
How did you conclude that the hang is related to a PCI bridge? I see recent PCI messages in the photo, but it looks like the last message is from NFS, so I'm wondering if I'm missing some context. The v6.0 dmesg shows several other ntfs, fuse, JFS, etc messages before more PCI-related things. Anyway, the "
initcall_debug" might help us narrow it down a bit.Thanks very much for the bisection (complete log at [1])!
The bisection claims the first bad commit is:
833477fce7a1("Merge tag 'sound-6.1-rc1' ofgit://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound")with parents:
7e6739b9336e("Merge tag 'drm-next-2022-10-05' ofgit://anongit.freedesktop.org/drm/drm")
86a4d29e7554("Merge tag 'asoc-v6.1' of https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound into for-linus")Both
7e6739b9336eand86a4d29e7554tested "good" during the bisection.There is a minor conflict when merging
86a4d29e7554into the upstream, but I can't imagine that being resolved incorrectly.
Quoted 7 lines
Would you mind turning off
CONFIG_SOUNDin your .config and testing833477fce7a1again? I'm a little skeptical that the hang would be sound-related, but I guess it's a place to start.Bjorn
Booting without sound into the commit 833477fce7a1 did not help. Same hang.
Best
Zeno
[+cc Krzysztof]
Quoted 24 lines · Zeno Davatz
Quoted 23 lines · Bjorn Helgaas
Quoted 22 lines · Bjorn Helgaas
Quoted 21 lines · Zeno Davatz
Quoted 19 lines · Bjorn Helgaas
Quoted 6 lines · On Wed, Dec 28, 2022 at 08:37:52AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216859
Summary: PCI bridge to bus boot hang at enumeration
Kernel Version: 6.1-rc1
...Quoted 6 lines
With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Same problem still exists with v6.1. and v6.2.-rc1
Thank you very much for your report, Zeno!
v6.0 works, v6.1-rc1 fails. Would you mind booting v6.1-rc1 with the "
ignore_loglevelinitcall_debug" kernel parameters and taking a photo when it hangs?
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
I will try this after Januar 7th 2023.
I updated the issue:
https://bugzilla.kernel.org/show_bug.cgi?id=216859
I booted with the option: "
ignore_loglevelinitcall_debug"
Thanks! There's so much pcie output in that picture that we can't see any of the initcall logging. Can you capture another movie, but use kernel parameters like "ignore_loglevel initcall_debug boot_delay=100" to slow things down? The full-speed boot is too fast for the camera to capture all the output. You can do this on any convenient kernel that hangs.
There might be more we can do with the bisection, too, but I don't have any suggestions for that yet. Maybe Krzysztof does.
Bjorn
Dear Bjorn
Quoted 77 lines · Bjorn Helgaas
[+cc [email protected] to avoid spamassassin]
Quoted 31 lines · Bjorn Helgaas
Quoted 30 lines · Zeno Davatz
Quoted 17 lines · Bjorn Helgaas
Quoted 16 lines · Bjorn Helgaas
Quoted 15 lines · Zeno Davatz
Quoted 14 lines · Bjorn Helgaas
Quoted 12 lines · On Wed, Dec 28, 2022 at 08:37:52AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216859
Summary: PCI bridge to bus boot hang at enumeration
Kernel Version: 6.1-rc1
...With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Same problem still exists with v6.1. and v6.2.-rc1
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Thank you very much for your report, Zeno!
v6.0 works, v6.1-rc1 fails. Would you mind booting v6.1-rc1 with the "
ignore_loglevelinitcall_debug" kernel parameters and taking a photo when it hangs?I will try this after Januar 7th 2023.
I updated the issue:
Quoted 8 lines
Quoted 1 line
I booted with the option: "
ignore_loglevelinitcall_debug"Thanks! There's so much pcie output in that picture that we can't see any of the initcall logging. Can you capture another movie, but use kernel parameters like "
ignore_loglevelinitcall_debugboot_delay=100" to slow things down? The full-speed boot is too fast for the camera to capture all the output. You can do this on any convenient kernel that hangs.Thanks for the new movie! The last initcalls I see before the hang are:
init_mqueue_fs
key_proc_init
jent_mod_initWe must have returned from
jent_mod_init()because I think the "saving config space" messages we see at the hang are frompcie_portdrv_init().I built
833477fce7a1("Merge tag 'sound-6.1-rc1' ofgit://git.kernel.org/pub/scl) with your .config and when I boot it on qemu, I see this:calling
jent_mod_init+0x0/0x32@ 1
initcalljent_mod_init+0x0/0x32returned 0 after 27185 usecs callingaf_alg_init+0x0/0x45@ 1
NET: RegisteredPF_ALGprotocol family
...
callingsg_pool_init+0x0/0xb4@ 1
initcallsg_pool_init+0x0/0xb4returned 0 after 462 usecs callingpcie_portdrv_init+0x0/0x43@ 1
pcieport 0000:00:1c.0: vgaarb:pci_notify
pcieport 0000:00:1c.0: runtime IRQ mapping not provided by arch pcieport 0000:00:1c.0: enabling bus mastering
pcieport 0000:00:1c.0: PME: Signaling with IRQ 24
pcieport 0000:00:1c.0: AER: enabled with IRQ 24
pcieport 0000:00:1c.0: saving config space at offset0x0(reading0x34208086) pcieport 0000:00:1c.0: saving config space at offset0x4(reading0x100507) pcieport 0000:00:1c.0: saving config space at offset0x8(reading0x6040002)
...
Quoted 5 lines
Would you mind trying again with "boot_delay=1000 pcie_ports=compat"?
"boot_delay=1000" should slow it down more (all the action is in the last 3 seconds and it's still hard to see) and "pcie_ports=compat" should turn off the PCIe port driver.
Done. Please see:
https://bugzilla.kernel.org/show_bug.cgi?id=216859#c42
Best
Zeno
[+cc [email protected] to avoid spamassassin]
Quoted 31 lines · Bjorn Helgaas
Quoted 30 lines · Zeno Davatz
Quoted 17 lines · Bjorn Helgaas
Quoted 16 lines · Bjorn Helgaas
Quoted 15 lines · Zeno Davatz
Quoted 14 lines · Bjorn Helgaas
Quoted 12 lines · On Wed, Dec 28, 2022 at 08:37:52AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216859
Summary: PCI bridge to bus boot hang at enumeration
Kernel Version: 6.1-rc1
...With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Same problem still exists with v6.1. and v6.2.-rc1
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Thank you very much for your report, Zeno!
v6.0 works, v6.1-rc1 fails. Would you mind booting v6.1-rc1 with the "
ignore_loglevelinitcall_debug" kernel parameters and taking a photo when it hangs?I will try this after Januar 7th 2023.
I updated the issue:
Quoted 8 lines
Quoted 1 line
I booted with the option: "
ignore_loglevelinitcall_debug"Thanks! There's so much pcie output in that picture that we can't see any of the initcall logging. Can you capture another movie, but use kernel parameters like "
ignore_loglevelinitcall_debugboot_delay=100" to slow things down? The full-speed boot is too fast for the camera to capture all the output. You can do this on any convenient kernel that hangs.
Thanks for the new movie! The last initcalls I see before the hang are:
init_mqueue_fs
key_proc_init
jent_mod_init
We must have returned from jent_mod_init() because I think the "saving config space" messages we see at the hang are from pcie_portdrv_init().
I built 833477fce7a1 ("Merge tag 'sound-6.1-rc1' ofgit://git.kernel.org/pub/scl) with your .config and when I boot it on qemu, I see this:
calling jent_mod_init+0x0/0x32 @ 1
initcall jent_mod_init+0x0/0x32 returned 0 after 27185 usecs calling af_alg_init+0x0/0x45 @ 1
NET: Registered PF_ALG protocol family
...
calling sg_pool_init+0x0/0xb4 @ 1
initcall sg_pool_init+0x0/0xb4 returned 0 after 462 usecs calling pcie_portdrv_init+0x0/0x43 @ 1
pcieport 0000:00:1c.0: vgaarb: pci_notify
pcieport 0000:00:1c.0: runtime IRQ mapping not provided by arch pcieport 0000:00:1c.0: enabling bus mastering
pcieport 0000:00:1c.0: PME: Signaling with IRQ 24
pcieport 0000:00:1c.0: AER: enabled with IRQ 24
pcieport 0000:00:1c.0: saving config space at offset 0x0 (reading 0x34208086) pcieport 0000:00:1c.0: saving config space at offset 0x4 (reading 0x100507) pcieport 0000:00:1c.0: saving config space at offset 0x8 (reading 0x6040002)
...
Would you mind trying again with "boot_delay=1000 pcie_ports=compat"?
"boot_delay=1000" should slow it down more (all the action is in the last 3 seconds and it's still hard to see) and "pcie_ports=compat" should turn off the PCIe port driver.
Bjorn
[+cc folks from 145eed48de27 and framebuffer folks, regression list]
Quoted 8 lines · Bjorn Helgaas
Quoted 7 lines · Bjorn Helgaas
Quoted 5 lines · On Wed, Dec 28, 2022 at 08:37:52AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216859
Summary: PCI bridge to bus boot hang at enumeration
Kernel Version: 6.1-rc1
...
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Same problem still exists with v6.1. and v6.2.-rc1
This is a regression between v6.0 and v6.1-rc1. Console output during boot freezes after nvidiafb deactivates the VGA console.
It was a lot of work for Zeno, but we finally isolated this console hang to 145eed48de27 ("fbdev: Remove conflicting devices on PCI bus").
The system actually does continue to boot and is accessible via ssh, but the console appears hung, at least for output. More details in the bugzilla starting at
https://bugzilla.kernel.org/show_bug.cgi?id=216859#c47 .
Bjorn
[+cc Geert]
Quoted 19 lines · Bjorn Helgaas
Quoted 8 lines · Bjorn Helgaas
Quoted 7 lines · Bjorn Helgaas
Quoted 5 lines · On Wed, Dec 28, 2022 at 08:37:52AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216859
Summary: PCI bridge to bus boot hang at enumeration
Kernel Version: 6.1-rc1
...Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Same problem still exists with v6.1. and v6.2.-rc1
This is a regression between v6.0 and v6.1-rc1. Console output during boot freezes after nvidiafb deactivates the VGA console.
Quoted 7 lines
It was a lot of work for Zeno, but we finally isolated this console hang to
145eed48de27("fbdev: Remove conflicting devices on PCI bus").The system actually does continue to boot and is accessible via ssh, but the console appears hung, at least for output. More details in the bugzilla starting at
https://bugzilla.kernel.org/show_bug.cgi?id=216859#c47 .
145eed48de27 ("fbdev: Remove conflicting devices on PCI bus") doesn't say what the benefit is, or what would break if we reverted it.
Does anybody have any clues? It would be nice to resolve this regression before v6.2, which will probably be released 2/12 or 2/19.
Bjorn
Quoted 21 lines · Bjorn Helgaas
[+cc folks from
145eed48de27and framebuffer folks, regression list]Quoted 8 lines · Bjorn Helgaas
Quoted 7 lines · Bjorn Helgaas
Quoted 5 lines · On Wed, Dec 28, 2022 at 08:37:52AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216859
Summary: PCI bridge to bus boot hang at enumeration
Kernel Version: 6.1-rc1
...Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Same problem still exists with v6.1. and v6.2.-rc1
This is a regression between v6.0 and v6.1-rc1. Console output during boot freezes after nvidiafb deactivates the VGA console.
Quoted 7 lines
It was a lot of work for Zeno, but we finally isolated this console hang to
145eed48de27("fbdev: Remove conflicting devices on PCI bus").The system actually does continue to boot and is accessible via ssh, but the console appears hung, at least for output. More details in the bugzilla starting at
https://bugzilla.kernel.org/show_bug.cgi?id=216859#c47 .
#regzbot introduced: 145eed48de27
#regzbot dup: https://lore.kernel.org/all/[email protected]/
Quoted 25 lines · Bjorn Helgaas
Quoted 24 lines · Bjorn Helgaas
[+cc folks from
145eed48de27and framebuffer folks, regression list]Quoted 8 lines · Bjorn Helgaas
Quoted 7 lines · Bjorn Helgaas
Quoted 5 lines · On Wed, Dec 28, 2022 at 08:37:52AM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=216859
Summary: PCI bridge to bus boot hang at enumeration
Kernel Version: 6.1-rc1
...Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
With Kernel 6.1-rc1 the enumeration process stopped working for me, see attachments.
The enumeration works fine with Kernel 6.0 and below.
Same problem still exists with v6.1. and v6.2.-rc1
This is a regression between v6.0 and v6.1-rc1. Console output during boot freezes after nvidiafb deactivates the VGA console.
It was a lot of work for Zeno, but we finally isolated this console hang to
145eed48de27("fbdev: Remove conflicting devices on PCI bus").
Quoted 7 lines
Quoted 4 lines
The system actually does continue to boot and is accessible via ssh, but the console appears hung, at least for output. More details in the bugzilla starting at
https://bugzilla.kernel.org/show_bug.cgi?id=216859#c47 .#regzbot introduced:
145eed48de27
#regzbot dup: https://lore.kernel.org/all/[email protected]/
#regzbot fix: 04119ab1a49f ("nvidiafb: detect the hardware support before removing console.")