Skip to contentSkip to conversation

Serial Console and Slow SCSI Disk Access?

DISCUSSION Jeff Gold 5 participants 9 messages Jun 12, 2006 – Jun 22, 2006

lore
Share & tools
lore

Jeff Gold

author
#0 lore raw

Does it make sense that enabling a serial console would reduce the bandwidth of a SCSI disk by more than a factor of ten? This seems crazy to me but there's no arguing with empirical facts. I have an IBM x345 system on which I installed an unmodified 2.6.16.20 kernel built with gcc-4.0.2-8.fc4 (the kernel configuration file I used can be found at http://augart.com/jgold/kconfig). With the control kernel command line I get about 70 MB/sec with hdparm -t but when I add "console=ttyS0,9600 console=tty0" I get about 1.6 MB/sec instead.

Here are the complete kernel command lines from grub.conf (the second one doesn't actually include a backslash in grub.conf -- that's just to avoid email wrapping):

ro root=/dev/sda3 pci=biosirq loglevel=7

  ro root=/dev/sda3 pci=biosirq console=ttyS0,9600n8 console=tty0 \
    loglevel=7

I have not been able to connect this behavior to any known kernel bugs using Google or mailing list archives. What might I be doing wrong?

Jeff

#1 lore raw
Quoted 1 line · Jeff Gold

Does it make sense that enabling a serial console would reduce the bandwidth of a SCSI disk by more than a factor of ten? This seems crazy to me but there's no arguing with empirical facts. I have an IBM x345 system on which I installed an unmodified 2.6.16.20 kernel built with gcc-4.0.2-8.fc4 (the kernel configuration file I used can be found at http://augart.com/jgold/kconfig). With the control kernel command line I get about 70 MB/sec with hdparm -t but when I add "console=ttyS0,9600 console=tty0" I get about 1.6 MB/sec instead.

This can happen if there are kernel messages being printed on the serial console. If all is quiet, I would expect things to be as fast as normal elsewhere.

-ml

author
#2 lore raw
Quoted 2 lines · Mark Lord

This can happen if there are kernel messages being printed on the serial console. If all is quiet, I would expect things to be as fast as normal elsewhere.

Thank you for the suggestion. I don't see much in /var/log/messages (syslogd is running). There are 3326 lines taking up about 256 kB there, and when I run hdparm runs no further messages are generated.

I don't have anything attached to the serial port at the moment. Could that cause problems? I'm going to attach something and see what happens. Other advice is still welcome.

Jeff

#3 lore raw
Quoted 7 lines · Jeff Gold
Quoted 2 lines · Mark Lord

This can happen if there are kernel messages being printed on the serial console. If all is quiet, I would expect things to be as fast as normal elsewhere.

Thank you for the suggestion. I don't see much in /var/log/messages (syslogd is running). There are 3326 lines taking up about 256 kB there, and when I run hdparm runs no further messages are generated.

I don't have anything attached to the serial port at the moment. Could that cause problems? I'm going to attach something and see what happens. Other advice is still welcome.

With nothing attached, any write to the serial device might go through a lengthy timeout because of flow control. I'd consider that a bug in this case though, and there is usually no console printout per scsi disk access either.

I would not be surprised if your serial console causes a longer boot time, as all boot messages have to be transferred over the slow serial link or in the worst case timed out one message at a time.

But I can't see why it'd make scsi disks slower. The scsi host adapter initialization writes some messages of course, but there should be no more console accesses during a hdparm test run.

Helge Hafting

author
#4 lore raw
Quoted 4 lines · Helge Hafting

With nothing attached, any write to the serial device might go
through a lengthy timeout because of flow control. [...] But I can't
see why it'd make scsi disks slower. The scsi host adapter initialization writes some messages of course, but there should be no more console accesses during a hdparm test run.

This makes sense to me. When I attach a serial cable and use that to login (I've got agetty running), hdparm produces no console messages that I can see using minicom. Still, the disk throughput is around 1.5 MB/sec for some reason. When I disable the serial console in grub.conf and reboot I get over 70 MB/sec again.

A combination of out-of-tree patches (mainly network related but also one to disable PM_TIMERS) seem to eliminate the issue even with the serial console enabled, at least for the moment. That means I no longer have a problem, but the whole thing is mysterious to me.

Jeff

#5 lore raw
Quoted 9 lines · Jeff Gold
Quoted 4 lines · Helge Hafting

With nothing attached, any write to the serial device might go
through a lengthy timeout because of flow control. [...] But I can't
see why it'd make scsi disks slower. The scsi host adapter initialization writes some messages of course, but there should be no more console accesses during a hdparm test run.

This makes sense to me. When I attach a serial cable and use that to login (I've got agetty running), hdparm produces no console messages that I can see using minicom. Still, the disk throughput is around 1.5 MB/sec for some reason. When I disable the serial console in grub.conf and reboot I get over 70 MB/sec again.

A combination of out-of-tree patches (mainly network related but also one to disable PM_TIMERS) seem to eliminate the issue even with the serial console enabled, at least for the moment. That means I no longer have a problem, but the whole thing is mysterious to me.

I can see one possibility, that I didn't think of yesterday.
Do the scsi host adapter share its interrupt with the serial line? (Boot a kernel that has the problem, and when scsi is slow, do a cat /proc/interrupts
If the scsi and the serial driver share an interrupt, then that is the source of the problem.

Linux can share interrupts without big performance problems - IF both of the hardware drivers are written with interrupt sharing in mind. Many linux drivers are, but some are not. So check if interrupt sharing is happening, and if so, contact the maintainers of your scsi driver and your serial port driver. Ask them if such sharing is ok or not. If you don't understand the /proc/interrupt listing, just send it to the maintainers so they can have a look.

Shared interrupts are to some extent set up by the bios, and to some extent by linux. So different kernel versions (or booting with a different set of drivers loaded, or a different distribution) may make a difference.

You can often get a pci device to use a different interrupt by moving it to another slot. That tends to solve interrupt sharing problems. I assume your scsi adapter is pci.

Helge Hafting

author
#6 lore raw
Quoted 2 lines · Helge Hafting

I can see one possibility, that I didn't think of yesterday. Do the scsi host adapter share its interrupt with the serial line?

That makes a lot of sense. I'll check it out and follow your instructions as soon as I can. Thanks.

Jeff

#7 lore raw
In reply to Helge Hafting
Quoted 25 lines · Helge Hafting
Quoted 9 lines · Jeff Gold
Quoted 4 lines · Helge Hafting

With nothing attached, any write to the serial device might go
through a lengthy timeout because of flow control. [...] But I can't
see why it'd make scsi disks slower. The scsi host adapter initialization writes some messages of course, but there should be no more console accesses during a hdparm test run.

This makes sense to me. When I attach a serial cable and use that to login (I've got agetty running), hdparm produces no console messages that I can see using minicom. Still, the disk throughput is around 1.5 MB/sec for some reason. When I disable the serial console in grub.conf and reboot I get over 70 MB/sec again.

A combination of out-of-tree patches (mainly network related but also one to disable PM_TIMERS) seem to eliminate the issue even with the serial console enabled, at least for the moment. That means I no longer have a problem, but the whole thing is mysterious to me.

I can see one possibility, that I didn't think of yesterday.
Do the scsi host adapter share its interrupt with the serial line? (Boot a kernel that has the problem, and when scsi is slow, do a cat /proc/interrupts
If the scsi and the serial driver share an interrupt, then that is the source of the problem.

Linux can share interrupts without big performance problems - IF both of the hardware drivers are written with interrupt sharing in mind. Many linux drivers are, but some are not. So check if interrupt sharing is happening, and if so, contact the maintainers of your scsi driver and your serial port driver. Ask them if such sharing is ok or not. If you don't understand the /proc/interrupt listing, just send it to the maintainers so they can have a look.

Quoted 8 lines

Shared interrupts are to some extent set up by the bios, and to some extent by linux. So different kernel versions (or booting with a different set of drivers loaded, or a different distribution) may make a difference.

You can often get a pci device to use a different interrupt by moving it to another slot. That tends to solve interrupt sharing problems. I assume your scsi adapter is pci.

I ran into a similar problem (extremely slow scsi performance), but hadn't reported it as I made a lot of changes at the same time (moved from 2.6.7 to 2.6.12.3 as well as enabling the serial console). but with this report I wanted to chime in with a weak 'me-too'. I haven't rebooted without the serial console yet to confirm this, but I will do so shortly, and try 2.6.17 to see if it has a similar problem or not.

David Lang

# cat /proc/interrupts
           CPU0       CPU1
  0:  458865186      29148    IO-APIC-edge  timer
  1:        216          7    IO-APIC-edge  i8042
  3:         78          1    IO-APIC-edge  serial
  4:         76          1    IO-APIC-edge  serial
  8:          8          1    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
 12:         84         10    IO-APIC-edge  i8042
 14:        332         10    IO-APIC-edge  ide0
 24:    4426414          1   IO-APIC-level  eth2
 26:  426930991          1   IO-APIC-level  eth0
 27:      17795  365692411   IO-APIC-level  eth1
 29:     553120         87   IO-APIC-level  ioc0
NMI:          0          0
LOC:  458903065  458904855
ERR:          0
MIS:          0
# lspci
0000:00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07)
0000:00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05)
0000:00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03)
0000:00:07.2 SMBus: Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0 (rev 02)
0000:00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05)
0000:00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
0000:00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
0000:00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12)
0000:00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01)
0000:00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 NorthBridge
0000:01:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
0000:01:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b)
0000:01:06.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
0000:02:02.0 PCI bridge: IBM PCI-X to PCI-X Bridge (rev 02)
0000:02:03.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5702X Gigabit Ethernet (rev 02)
0000:02:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5702X Gigabit Ethernet (rev 02)
0000:03:04.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
0000:03:04.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
0000:03:06.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
0000:03:06.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (rev 01)
0000:04:01.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev 07)
#8 lore raw
Quoted 20 lines · David Lang
Quoted 18 lines · Helge Hafting
Quoted 17 lines · Jeff Gold
Quoted 5 lines · Helge Hafting

With nothing attached, any write to the serial device might go through a lengthy timeout because of flow control. [...] But I can't see why it'd make scsi disks slower. The scsi host adapter initialization writes some messages of course, but there should be no
more console accesses during a hdparm test run.

This makes sense to me. When I attach a serial cable and use that to login (I've got agetty running), hdparm produces no console messages that I can see using minicom. Still, the disk throughput is around 1.5 MB/sec for some reason. When I disable the serial console in grub.conf and reboot I get over 70 MB/sec again.

A combination of out-of-tree patches (mainly network related but also one to disable PM_TIMERS) seem to eliminate the issue even with the serial console enabled, at least for the moment. That means I no longer have a problem, but the whole thing is mysterious to me.

Quoted 6 lines
Quoted 6 lines

I can see one possibility, that I didn't think of yesterday.
Do the scsi host adapter share its interrupt with the serial line? (Boot a kernel that has the problem, and when scsi is slow, do a cat /proc/interrupts
If the scsi and the serial driver share an interrupt, then that is the source of the problem.

[skipp]

Quoted 6 lines

I ran into a similar problem (extremely slow scsi performance), but hadn't reported it as I made a lot of changes at the same time (moved from 2.6.7 to 2.6.12.3 as well as enabling the serial console). but with this report I wanted to chime in with a weak 'me-too'. I haven't rebooted without the serial console yet to confirm this, but I will do so shortly, and try 2.6.17 to see if it has a similar problem or not.

2.6.17.1 here, MB Intel SCB20, aic7899 onboard. aic7xxx compilled in, serial core compilled in, serial console enabled.
With/without serial sonsole - speed 66.0 Mb/s.

Maybe problem related to MB chipset?