dropbear & openssh transfers are limited to about 800k !?

Hi!

I'm using buildroot revision 23203. After all seems to work on my at91sam9263 based board, I figured out that scp transferes not more than 800kByte in one file. This is not acceptable and I tried openssh instead of dropbear. Still, the maximum transferable file size is less than 800k.

Any idea where the limit is bound to?

Regards
akuehn

Andreas> Hi!

Andreas> I'm using buildroot revision 23203. After all seems to work Andreas> on my at91sam9263 based board, I figured out that scp Andreas> transferes not more than 800kByte in one file. This is not Andreas> acceptable and I tried openssh instead of dropbear. Still, Andreas> the maximum transferable file size is less than 800k.

Works here:

scp binutils-2.18.tar.bz2 root at 150.158.210.109:/tmp/ root at 150.158.210.109's password:
binutils-2.18.tar.bz2 100% 14MB 2.0MB/s 00:07

Andreas> Any idea where the limit is bound to?

Maybe you're running out of disk (flash/ramdisk) space?

Bye, Peter Korsgaard

Hey Peter!
Fine hearing that it should work as expected.

About the memory capacity:
Mem: 8612K used, 118892K free, 0K shrd, 320K buff, 5168K cached
rootfs: 119.5M used, 56.0M free

I still expect something like that ext2-fs has some limitation or uClibc has some buffer length to be adjusted. But if not, it sounds like my buildroot needs some "make dirclean" and hopefully all the quirks disappear. Any other ideas?

Not knowing the cause is somewhat uncomfortable. (I hate keeping fingers crossed. Last time I dropped my coffee pot!)

Regards
akuehn

Here too.

Try running scp and/or dropbear on target under strace or better gdbserver.
Does it also limit file size if you transfer file from target to host or only other way around?

Regards,
Hinko

Hinko Ko?evar, OSS developer
?ETRTA POT, d.o.o.
Planina 3, 4000 Kranj, SI EU
tel     ++386 (0) 4 280 66 03
e-mail  hinko.kocevar at cetrtapot.si
http    www.cetrtapot.si

First, scp (sftp) segfaults in both directions.

- getting a file from the host to the box ends at 880KB
- putting a file from the box to the host ends at 992KB

1. Here is the debug output of "scp user at host:gimp.ps ."
Right in the middle you see a "segmentation fault"

2. Further down you can find the not very enlighting result from
"strace scp user at host:gimp.ps . "

3. I tried it the opposite way. Sitting on the host and scp a file
from the box. The result and -vvv output is even more scary!

Well, summing up all, it looks as there are different aspects interfering each other but all of it is around the openssh system. Configuration is one part but currently I redo the buildroot system with my current configuration to get rid of leftovers from previous configurations. (That'll take a while...)
Nevertheless, can someone read tea leaves from the below?

Regards
akuehn

Output of "scp user at host:gimp.ps ."

Sending file modes: C0644 4146862 gimp.ps
debug2: channel 0: written 42 to efd 6
Sink: C0644 4146862 gimp.ps
gimp.ps 0% 0 0.0KB/s
--:-- ETAdebug2: channel 0: window 65472 sent adjust 65600
debug2: channel 0: window 49152 sent adjust 81920
debug2: channel 0: window 49152 sent adjust 81920
* akuehn: this message comes multiple times *
debug2: channel 0: window 49152 sent adjust 81920
gimp.ps 22% 896KB 896.0KB/s
00:01 debug2: channel 0: read<=0 rfd 4 len 0
debug2: channel 0: read failed
debug2: channel 0: close_read
debug2: channel 0: input open -> drain
debug2: channel 0: write failed
debug2: channel 0: close_write
debug2: channel 0: output open -> closed
debug2: channel 0: ibuf empty
debug2: channel 0: send eof
debug2: channel 0: input drain -> closed
debug2: channel 0: window 49152 sent adjust 65536 Segmentation fault

debug2: channel 0: window 49152 sent adjust 65536
debug2: channel 0: window 49152 sent adjust 65536
debug2: channel 0: window 49152 sent adjust 65536
* akuehn: this message comes multiple times *

debug2: channel 0: window 49152 sent adjust 65536
debug2: channel 0: window 49152 sent adjust 65536
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug2: channel 0: rcvd eof
debug2: channel 0: rcvd close
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cfd -1)

debug3: channel 0: close_fds r -1 w -1 e 6 c -1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 42 seconds
debug1: Bytes per second: stdin 00, stdout 00, stderr 00
debug1: Exit status 1

strace output "strace scp user at host:gimp.ps . "

read(7, "-!s8W-!s8W-!s8W-!s8W-!s8N\'!zz!!*"..., 4096) = 4096 write(3, "-!s8W-!s8W-!s8W-!s8W-!s8N\'!zz!!*"..., 4096) = 4096 read(7, "W-!s8W-!s8W-!s8W-!s8W-!s8W-!s8W-"..., 4096) = 4096 write(3, "W-!s8W-!s8W-!s8W-!s8W-!s8W-!s8W-"..., 4096) = 4096 read(7, "/\\;FNo=3&s;b3C6/$S=5uJs8W-!s8W-!"..., 4096) = 4096 write(3, "/\\;FNo=3&s;b3C6/$S=5uJs8W-!s8W-!"..., 4096) = 4096 --- SIGALRM (Alarm clock) @ 0 (0) ---

getpgrp()                               = 884
ioctl(1, TIOCGPGRP, [884])              = 0
time(NULL)                              = 6388

gimp.ps 8% 324KB 324.0KB/s
00:01 ) = 80
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++

Doing "scp user at box:openocd ."
...

debug3: Ignored env LESSCLOSE
debug3: Ignored env _
debug1: Sending command: scp -v -f openocd
debug2: channel 0: request exec confirm 0
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel 0: rcvd adjust 131072
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug2: channel 0: obuf empty
debug2: channel 0: close_write
debug2: channel 0: output drain -> closed
debug1: client_input_channel_req: channel 0 rtype exit-signal reply 0
debug2: channel 0: rcvd close
debug2: channel 0: close_read
debug2: channel 0: input open -> closed
debug3: channel 0: will not send data after close
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cfd -1)

debug3: channel 0: close_fds r -1 w -1 e 7 c -1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 0.0 seconds
debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
debug1: Exit status -1

I would suggest to run the segfaulting process under gdb(server) and backtrace when segfault occurs. You'll probably need to rebuild uClibc and busybox with debugging symbols.

...

Hinko Ko?evar, OSS developer
?ETRTA POT, d.o.o.
Planina 3, 4000 Kranj, SI EU
tel     ++386 (0) 4 280 66 03
e-mail  hinko.kocevar at cetrtapot.si
http    www.cetrtapot.si

Hi there...

it took a while but I did a complete rebuild of my buildroot environment. And for sure, I tried both versions of the uClibc. Finally I did an strace on dropbear and got these news:

% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------

   7676     0011853          72       164         1 read
   1256     0001940        1940         1           execve
   1068     0001649          14       118           write
    000     0000000           0         1           fork
    000     0000000           0        12         2 open
    000     0000000           0        13           close
    000     0000000           0         3           time
    000     0000000           0         1           alarm
    000     0000000           0         3           pipe
    000     0000000           0         4           brk
    000     0000000           0         8         2 ioctl
    000     0000000           0         2           umask
    000     0000000           0         2           getpgrp
    000     0000000           0         4           munmap
    000     0000000           0         1           stat
    000     0000000           0         7           fstat
    000     0000000           0         4           mprotect
    000     0000000           0         6           rt_sigaction
    000     0000000           0        19           mmap2
    000     0000000           0         2           stat64
    000     0000000           0         1           getuid32

------ ----------- ----------- --------- --------- ----------------
100.00 0015442 376 5 total

To me it looks like uClibc calls are not to happy?!
To keep this posting as short as possible I added some *interesting* sniplets from a normal strace run. Does that give somebody a clue about the problem? Is that a configuration issue?

    ...

brk(0x3e000) = 0x3e000
getuid32() = 0
open("/etc/passwd", O_RDONLY) = 3
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbedec990) = -1 ENOTTY (Inappropriate ioctl for device)
read(3, "root:x:0:0:root:/root:/bin/sh\ndae"..., 4096) = 562
close(3) = 0
ioctl(2, SNDCTL_TMR_TIMEBASE or TCGETS, {B115200 opost isig icanon echo ...}) = 0
rt_sigaction(SIGPIPE, {0x16c9c, [PIPE], SA_RESTART|0x4000000}, {SIG_DFL, [], 0}, 8) = 0
pipe([3, 4]) = 0
pipe([5, 6]) = 0

... and finally

write(3,
"W\260\23\341\246\330\v\tU\34\202\302b at JH\275CY`ywt\264\r\213D\264\331\31\267}m"..., 4096) = 4096
read(7, "\3V\226\226\340\337R\261\270\214\317\253+\370,\336,\225\227K\213\177)\225J\345\322\352\322\352R\251\370"..1
read(7, 0x3d1af, 25) = ? ERESTARTSYSTo be restarted)
--- SIGALRM (Alarm clock) @ 0 (0) ---

getpgrp()                               = 826
ioctl(1, TIOCGPGRP, [826])              = 0
time(NULL)                              = 2444

write(1, "\rbigPayLoad "..., 80) = 80
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++

  aKuehn

--------------> Never trust a computer you can't lift. -- Stan Masor

Okay, I did a complete rebuild of buildroot with the same configuration as befor. The transfer limitation is still there. Unfortunately, a have no alternative host system to check if the hosts configuration is what causes the trouble.
Host: i686/Debian Edge; SSH-2.0-OpenSSH_4.3p2 Debian-9etch3 Box : at91sam9263/buildroot; SSH-2.0-OpenSSH_4.6

Questions, Ideas?

Regards
akuehn

Hi,

Andreas> Fine hearing that it should work as expected.

Andreas> About the memory capacity:
Andreas> Mem: 8612K used, 118892K free, 0K shrd, 320K buff, 5168K cached Andreas> rootfs: 119.5M used, 56.0M free

Andreas> I still expect something like that ext2-fs has some Andreas> limitation or uClibc

Ext2? Are you using an initrd? How much space do you have free?

Bye, Peter Korsgaard

Good Morning Peter!

No, I do *not* use initrd.

The rootfs is an ext2 on a CompactFlash card (Industial Type). I tried cards from different brands but without result. Moreover, Linux uses these cards through its IDE interface. I can copy 5MB files around the disk without trouble. To me, the storage system works fine.