/dev/random problems _not_ solved in 2.6.26
Hi,
this is just to let you know that the /dev/random problems are still not solved in the 2.6.26 version. A small dictionary attack on sshd (in my case 260 tries over 12 minutes) are able to produce
auth.log.0:Aug 3 05:30:35 dirk sshd[1825]: fatal: Couldn't obtain random bytes (error 604389476)
and the sshd dies (followed by apache on the first subsequent SSL request and so on).
Unfortunately I can't help you in tracing the uml process or trying patches - this is a virtual server at my provider and I don't have any access there. I could try to get the .config file.
# uname -a Linux dirk 2.6.26 #1 Wed Jul 30 10:56:10 CEST 2008 i686 GNU/Linux
# cat /proc/cpuinfo processor : 0 vendor_id : User Mode Linux model name : UML mode : skas host : Linux seldon-base 2.6.23.17 #2 SMP Sat Jun 21 15:04:22 CEST 2008 i686 bogomips : 4771.02
# cat /proc/sys/kernel/random/entropy_avail 293
After exhausting the entropy with
cat /dev/random | od -cx
the entropy only slooowly gets up, one byte per several seconds or so with me typing in the ssh console...
Regards
Signature · 12 lines
Stano ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ User-mode-linux-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Quoted 10 lines · Stanislav Meduna
this is just to let you know that the
/dev/randomproblems are still not solved in the 2.6.26 version. A small dictionary attack on sshd (in my case 260 tries over 12 minutes) are able to produceauth.log.0:Aug 3 05:30:35 dirk sshd[1825]: fatal: Couldn't obtain random bytes (error 604389476)
and the sshd dies (followed by apache on the first subsequent SSL request and so on).
Off the top of my head, you need the following:
2.6.26 UML, which you do
rngtools installed in the UML
sufficient entropy on the host
This last one shouldn't be taken for granted. I had problems with this when debugging the random driver changes - on my laptop with me typing on it, mousing, disk going, and network traffic going in and out. It would be worse on a colocated, no-keyboard, no-mouse, server, which are notorious for this problem.
Quoted 4 lines
Unfortunately I can't help you in tracing the uml process or trying patches - this is a virtual server at my provider and I don't have any access there. I could try to get the .config file.
You could ask them about how their servers get their randomness.
If nothing else works, I suppose a nasty workaround might be to replace /dev/random with dev/urandom, but that has security implications.
Jeff
Signature · 11 lines
Work email - jdike at linux dot intel dot com ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ User-mode-linux-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Quoted 3 lines · Jeff Dike
If nothing else works, I suppose a nasty workaround might be to replace
/dev/randomwith dev/urandom, but that has security implications.
I am aware of the implications, but unfortunately it did not help either.
# ls -l /dev/*random* crw-rw-rw- 1 root root 1, 8 2008-08-05 17:41 /dev/oldrandom lrwxrwxrwx 1 root root 7 2008-08-05 17:41 /dev/random -> urandom crw-rw-rw- 1 root root 1, 9 2008-08-05 17:42 /dev/urandom
# grep random /var/log/auth.log Aug 5 19:53:50 dirk sshd[3117]: fatal: Couldn't obtain random bytes (error 604389476)
# cat /proc/sys/kernel/random/entropy_avail 335
I am now talking to the host owner - he is running quite a few UML guests, each in chroot environment, and one of the problems is that his entropy gets sucked out...
Now he has replaced random with urandom in my chroot jail, so the UML's random is his urandom. Let's see what happens...
Signature · 12 lines
Stano ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ User-mode-linux-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Quoted 16 lines
-----Original Message-----
I am aware of the implications, but unfortunately it did not help either.# ls -l /dev/*random* crw-rw-rw- 1 root root 1, 8 2008-08-05 17:41 /dev/oldrandom lrwxrwxrwx 1 root root 7 2008-08-05 17:41 /dev/random -> urandom crw-rw-rw- 1 root root 1, 9 2008-08-05 17:42 /dev/urandom# grep random /var/log/auth.log Aug 5 19:53:50 dirk sshd[3117]: fatal: Couldn't obtain random bytes (error 604389476)# cat /proc/sys/kernel/random/entropy_avail 335
Quoted 6 lines
I am now talking to the host owner - he is running quite a few UML guests, each in chroot environment, and one of the problems is that his entropy gets sucked out...
Now he has replaced random with urandom in my chroot jail, so the UML's random is his urandom. Let's see what happens...
Please keep me appraised of what you find. Our servers never had this problem until upgrading to a 2.6.2x version of the kernel. SSHD, Apache and several other software packages worked fine under previous versions (and still do when we downgrade them). Its also worth noting that this behavior has never been exhibited on the host system. We've only seen it on our virtual systems.
I keep meaning to go back and search for exactly which version introduced the changed behavior, but I don't have time. Instead we now have scripts that automatically restart processes that use OpenSSL when they die. We tried shifting to urandom earlier to see if it would resolve the issue, but didn't see a change in behavior. I'm interested in knowing if your approach will function better.
Tony
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Quoted 4 lines · Anthony - NET
Quoted 2 lines
Now he has replaced random with urandom in my chroot jail, so the UML's random is his urandom. Let's see what happens...
Please keep me appraised of what you find.
Look like this works - at least there were no problems over the night.
Quoted 3 lines
Instead we now
have scripts that automatically restart processes that use OpenSSL when they die.
Tried this too. Note that if you are checking the affected ports it can actually make the situation worse - I tried this using monit checking for ssh and https and as it creates a SSL connection to do this, it needs the entropy, both the client and the server.
Quoted 2 lines
We tried shifting to urandom earlier to see if it would resolve the issue, but didn't see a change in behavior.
I looked into the openssl code (vanilla, dunno whether there are OS-specific patches, I am using Debian testing).
It tries to use both /dev/urandom, /dev/random and /dev/srandom, and it filters them according to major/minor - so linking the /dev/* on the guest won't help, it picks one. However, this one should be the urandom first...
It then select-s(!) for the device, 10 ms in each step. If anything fails, it proceeds to the next device and tries the same there too.
There is a suspicious code sequence:
if (select(fd+1,&fset,NULL,NULL,&t) >= 0)
...
usec = t.tv_usec;
...
/* Some Unixen will update t in select(), some
won't. For those who won't, or if we
didn't use select() in the first place,
give up here, otherwise, we will do
this once again for the remaining
time. */
if (usec == 10*1000)
usec = 0;
From the static code inspection it looks like one
of the following situations on urandom produces this:
- select returns -1 with errno other than EINTR/EAGAIN
- read on the device returns -1 or (!!!) 0 with errno other than EINTR/EAGAIN
- select select's successfully and immediately, leaves the time not slept unchanged in the time argument (which is IMHO fully legal, if it finds the bytes immediately) _and_ the read does not get all the needed bytes.
I don't have time now to exactly debug what is wrong, but at least the last case looks as a bug in openssl, triggered by some change in the select or read-behaviour. I'll contact the openssl developers about it.
Linking the random to urandom on the host should make the situation better, as it gives the guest a chance for trying "another" device...
Regards
Signature · 11 lines
Stano ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ User-mode-linux-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Re: [uml-devel] /dev/random problems .. or FP registers corruption?!
Hi,
Well, I am now puzzled quite a bit more. Another problem on this UML box is that if I do tail -f on some file, I ocassionally get an assertion:
tail: xnanosleep.c:67: xnanosleep: Assertion `0 <= seconds' failed.
This seems to happen more often (or at all) when I am doing something using the openssl - tail -f on an apache access log runs fine when I am doing http accesses and fails quite repeatedly when I am doing https or starting/terminating new SSL connection from/to the box.
I also have a few strace-s from the failing sshd, where I can't really understand how the code got there.
These two have one thing in common - a double variable that got initialized long ago and at the time of the problem it looks like it mysteriously changed value:
"entropy" in the case of openssl and "sleep_interval" in tail.
Am I seeing ghosts? Anyone got other mysterious problems with current UML kernel? Could it be that some state-saving method is corrupting fp registers or something like that? Was there some change in the UML / vanilla kernel recently?
To the openssl-ers: I am starting to think that openssl is fine here and that the problem is indeed in the UML kernel.
Regards
Signature · 11 lines
Stano ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ User-mode-linux-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Re: [uml-devel] /dev/random problems .. or FP registers corruption?!
Quoted 5 lines · Stanislav Meduna
Am I seeing ghosts? Anyone got other mysterious problems with current UML kernel? Could it be that some state-saving method is corrupting fp registers or something like that? Was there some change in the UML / vanilla kernel recently?
Confirmed: I can reproduce openssl changing a double value in unrelated process. I was not able to come up with a code not relying on external software yet:
===
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
#include <time.h>
#include <signal.h>
testdblchg()
{
volatile double x0 = 2.0;
volatile double x1;
int iter = 0;
fprintf(stderr, "Starting loop, pid=%d\n", getpid());
while(1)
{
++iter;
x1 = x0;
x0 += 1.0;
if (x0 != x1 + 1)
{
kill(getppid(), SIGTERM);
fprintf(stderr, "Double variable changed! pid=%d, iteration=%d, should=%g,
is=%g\n", getpid(), iter, x1+1, x0);
exit(1);
}
sleep(1); }
}
testcrypto()
{
while(1)
{
system("openssl genrsa -out /dev/null 4096");
sleep(1); }
}
main()
{
if (fork() == 0)
testdblchg();
testcrypto();
}
===
Output:
$ ./a.out Starting loop, pid=16920 Generating RSA private key, 4096 bit long modulus ......++ .......................................................................................................................................................................++ e is 65537 (0x10001) Generating RSA private key, 4096 bit long modulus ...................................Double variable changed! pid=16920, iteration=15, should=nan, is=nan Terminated
$ uname -a Linux dirk 2.6.26 #1 Wed Jul 30 10:56:10 CEST 2008 i686 GNU/Linux
$ cat /proc/cpuinfo processor : 0 vendor_id : User Mode Linux model name : UML mode : skas host : Linux seldon-base 2.6.23.17 #2 SMP Sat Jun 21 15:04:22 CEST 2008 i686 bogomips : 4771.02
$ gcc -v Using built-in specs. Target: i486-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Debian 4.3.1-2' --with-bugurl=file:///usr/share/doc/gcc-4.3/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.3 --program-suffix=-4.3 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --enable-targets=all --enable-cld --enable-checking=release --build=i486-linux-gnu --host=i486-linux-gnu --target=i486-linux-gnu Thread model: posix gcc version 4.3.1 (Debian 4.3.1-2)
$ dpkg -l libc6 Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad) ||/ Name Version Description +++-==============-==============-============================================ ii libc6 2.7-10 GNU C Library: Shared libraries
Regards
Signature · 11 lines
Stano ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ User-mode-linux-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
[uml-devel] FP registers corruption
Erm.. after debugging the FP corruption for several hours trying to isolate the cause I stumbled upon a patch from Jeff Dike from February:
http://article.gmane.org/gmane.linux.uml.devel/11412
Did this or equivalent patch get into the mainline kernel? Browsing through the source I don't think so :( If not, where is the best place to fetch the UML tree with the patches that the developers seem to find necessary, applied?
Thanks
Signature · 12 lines
Stano ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ User-mode-linux-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Re: [uml-devel] FP registers corruption
Quoted 11 lines · Stanislav Meduna
Erm.. after debugging the FP corruption for several hours trying to isolate the cause I stumbled upon a patch from Jeff Dike from February:
http://article.gmane.org/gmane.linux.uml.devel/11412
Did this or equivalent patch get into the mainline kernel? Browsing through the source I don't think so :( If not, where is the best place to fetch the UML tree with the patches that the developers seem to find necessary, applied?
It's there. I just eyeballed the source. It's commit 2f56debd77a8f52f1ac1d3c3d89cc7ce5e083230 - the only way you wouldn't have it is by having a too-old version of UML.
Jeff
Signature · 11 lines
Work email - jdike at linux dot intel dot com ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ User-mode-linux-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Re: [uml-devel] FP registers corruption
Quoted 9 lines · Jeff Dike
Quoted 5 lines
Did this or equivalent patch get into the mainline kernel? Browsing through the source I don't think so :( If not, where is the best place to fetch the UML tree with the patches that the developers seem to find necessary, applied?
It's there. I just eyeballed the source. It's commit
2f56debd77a8f52f1ac1d3c3d89cc7ce5e083230- the only way you wouldn't have it is by having a too-old version of UML.
Yes, you're right - sorry. I probably looked at the wrong version (I used some diff-ing web interface) :( However, the problem is still here.
I tried to isolate the exact version, but I was not able to compile/run all the UML versions in my test environment (current Ubuntu) - obviously the UML is quite sensitive to the host kernel as well.
So the only thing I can say is that I cannot reproduce it with neither 2.6.24.7 nor 2.6.26.2 guest in my environment. The only instance I am seeing it is the 2.6.26 guest on 2.6.23.17 host at my provider :(
The owner of the host machine where the problem is easily reproducible is back from the vacation, so I hope to have some more information soon; however I am not sure I will be able to convince him to update the host kernel too (that means scheduled downtime etc., ...).
Any hint what should I grep and code-review for? The symptoms of the bug look very similar to the page-fault induced one, so the cause is maybe also similar/the same. I played with the scheduler (sched_yield) etc and it does not seem that it is the 'normal' context switch who is corrupting the registers, more like it is some interrupt handler or something like that.
Any other folks seeing ssh or spache/ssl crashing: could you please post the UML and host versions and any other relevant informatio such as number of the UML kernels running, whether the machine is under memory pressure, devices used (e.g. heavy networking usage), ...?
Thanks
Signature · 11 lines
Stano ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ User-mode-linux-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Re: [uml-devel] FP registers corruption
Quoted 4 lines · Stanislav Meduna
I tried to isolate the exact version, but I was not able to compile/run all the UML versions in my test environment (current Ubuntu) - obviously the UML is quite sensitive to the host kernel as well.
It's not. Any 2.6 UML should run on a any 2.6 host. If you are having problems that look like version skew problems, say what they are.
Quoted 4 lines
So the only thing I can say is that I cannot reproduce it with neither 2.6.24.7 nor 2.6.26.2 guest in my environment. The only instance I am seeing it is the 2.6.26 guest on 2.6.23.17 host at my provider :(
In an earlier email, you provide a nice test case. However, I'm running that here, and getting page after page of successful key generation.
Is there anything else that might be relevant to reproducing this?
Jeff
Signature · 11 lines
Work email - jdike at linux dot intel dot com ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ User-mode-linux-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Re: [uml-devel] FP registers corruption
Quoted 3 lines · Jeff Dike
It's not. Any 2.6 UML should run on a any 2.6 host. If you are having problems that look like version skew problems, say what they are.
My host is current Ubuntu and I was unable to test 2.6.25.x there due to
Locating the top of the address space ... Address 0x0 no good?
which AFAIK was a known problem caused by some ubuntu-specific patch.
2.6.26 crashes here with
Locating the bottom of the address space ... 0x10000
Locating the top of the address space ... Checking that ptrace can change system call numbers...0xc0000000
Core dump limits :
soft - 0
hard - NONE
OK
Checking syscall emulation patch for ptrace...0xc0000000
Core dump limits :
soft - 0
hard - NONE
OK
Checking advanced syscall emulation patch for ptrace...0xc0000000
Core dump limits :
soft - 0
hard - NONE
OK0xc0000000
Core dump limits :
soft - 0
hard - NONE
Checking for tmpfs mount on /dev/shm...OK
Checking PROT_EXEC mmap in /dev/shm/...OK
Checking for the skas3 patch in the host:
- /proc/mm...not found: No such file or directory
- PTRACE_FAULTINFO...not found
OK
- PTRACE_LDT...not found
OK
OK
UML running in SKAS0 mode
Linux version 2.6.26 (stano@maroon) (gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)) #2 Mon Aug 11 23:50:44 CEST 2008
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 6096 Kernel command line: root=/dev/ubda ubd0=/home/stano/uml/test.rootfs mem=24M con=pty con0=fd:0,fd:1 con1=xterm eth0=daemon,,unix,/var/run/uml-utilities/uml_switch.ctl PID hash table entries: 128 (order: 7, 512 bytes)
Dentry cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 2048 (order: 1, 8192 bytes)
Memory: 21684k available
SLUB: Genslabs=12, HWalign=128, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 Mount-cache hash table entries: 512
Checking for host processor cmov support...Yes
Checking that host ptys support output SIGIO...Yesnet_namespace: 624 bytes
NET: Registered protocol family 16
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 1024 (order: 1, 8192 bytes)
TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
TCP: Hash tables configured (established 1024 bind 1024)
TCP reno registered
NET: Registered protocol family 1
mconsole (version 2) initialized on /home/stano/.uml/umltest/mconsole Checking host MADV_REMOVE support...OK
ubd: Synchronous mode
UML Watchdog Timer
Host TLS support detected
Detected host type: i386 (GDT indexes 6 to 9)
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
msgmni has been set to 42
io scheduler noop registered
io scheduler anticipatory registered (default)
io scheduler deadline registered
io scheduler cfq registered
brd: module loaded
loop: module loaded
SoftDog: cannot register miscdev on minor=130 (err=-16)
Mirror/redirect action on
TCP cubic registered
NET: Registered protocol family 17
Initialized stdio console driver
Console initialized on /dev/tty0
console [tty0] enabled
Initializing software serial port version 1
Choosing a random ethernet address for device eth0
Netdevice 0 (9a:fd:f3:b2:aa:10) : daemon backend (uml_switch version 3) - unix:/var/run/uml-utilities/uml_switch.ctl
console [mc-1] enabled
ubda: unknown partition table
VFS: Mounted root (ext2 filesystem) readonly.
Stub registers -
0 - 0
1 - 0
2 - 0
3 - 0
4 - 0
5 - 0
6 - 0
7 - 0
8 - 0
9 - 0
10 - 0
11 - ffffffff
12 - 0
13 - 73
14 - 10202
15 - 0
16 - 7b
wait_stub_done : failed to wait for SIGTRAP, pid = 24625, n = 24625, errno = 25, status =
0xb7f
Kernel panic - not syncing: Attempted to kill init!EIP: 0073:[<08189151>] CPU: 0 Not tainted ESP: 007b:bfb22dc0 EFLAGS: 00000246
Not tainted
EAX: 00000000 EBX: 0000602c ECX: 00000013 EDX: 0000602c
ESI: 00006028 EDI: 0828e818 EBP: bfb2379a DS: 007b ES: 007b
09463bd8: [<0808322b>] notifier_call_chain+0x25/0x48 09463bfc: [<08083283>] atomic_notifier_call_chain+0x15/0x19 09463c14: [<0807011b>] panic+0x52/0xce 09463c30: [<08072ec0>] do_exit+0x5c/0x5bf 09463c4c: [<08078767>] recalc_sigpending+0x16/0x25 09463c64: [<0807349c>] do_group_exit+0x79/0xa5 09463c7c: [<0807af6d>] get_signal_to_deliver+0x286/0x29b 09463ca0: [<08058dc6>] do_signal+0x230/0x2e7 09463d00: [<0807961e>] send_signal+0x197/0x1ae 09463d1c: [<08065a5a>] set_signals+0x18/0x28 09463d24: [<080799f7>] force_sig_info+0x9f/0xa9 09463d44: [<08079a10>] force_sig+0xf/0x13 09463d68: [<0805a26d>] fatal_sigsegv+0x1b/0x20 09463d78: [<08067df1>] wait_stub_done+0xa1/0xa7 09463da4: [<08067681>] do_syscall_stub+0x9c/0x16b 09463dc4: [<08068d6d>] write_ldt_entry+0xef/0x14f 09463e0c: [<08069482>] init_new_ldt+0x193/0x2d5 09463e2c: [<080935b5>] __alloc_pages_internal+0x7d/0x341 09463e88: [<0805a6e3>] init_new_context+0xc7/0x108 09463ea4: [<080ade0f>] bprm_mm_init+0x41/0x169 09463ed0: [<080aea02>] do_execve+0x6b/0x19a 09463ed8: [<080a9114>] do_sys_open+0x96/0xba 09463eec: [<0805738d>] execve1+0x21/0x39 09463f0c: [<080573f5>] um_execve+0x14/0x33 09463f14: [<0805907a>] kernel_execve+0x2b/0x34 09463f30: [<080570d5>] run_init_process+0x15/0x19 09463f40: [<0805717e>] init_post+0xa5/0xcf 09463f48: [<080485ab>] kernel_init+0x1e5/0x1ef 09463f58: [<080553b6>] tcp_congestion_default+0x0/0xc 09463fbc: [<08064cd4>] run_kernel_thread+0x38/0x3f 09463fd8: [<08064cb7>] run_kernel_thread+0x1b/0x3f 09463fe4: [<0805838e>] new_thread_handler+0x53/0x79 09463fe8: [<080483c6>] kernel_init+0x0/0x1ef
2.6.26.2 has no problems.
Quoted 3 lines
In an earlier email, you provide a nice test case. However, I'm running that here, and getting page after page of successful key generation.
Same here with 2.6.26.2 :(
Quoted 1 line
Is there anything else that might be relevant to reproducing this?
Well - the difference to my test environment is that
- the host machine is a SMP one (I am running the host inside vmware)
- there are quite a few UML guests running (AFAIK 5 to 8 - no space for testing this here yet)
Config should be the same, I used /proc/config.gz from the failing machine.
dirk:~# uname -a Linux dirk 2.6.26 #1 Wed Jul 30 10:56:10 CEST 2008 i686 GNU/Linux dirk:~# cat /proc/cpuinfo processor : 0 vendor_id : User Mode Linux model name : UML mode : skas host : Linux seldon-base 2.6.23.17 #2 SMP Sat Jun 21 15:04:22 CEST 2008 i686 bogomips : 4771.02
Signature · 11 lines
Stano ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ User-mode-linux-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Re: [uml-devel] FP registers corruption
Quoted 5 lines · Stanislav Meduna
My host is current Ubuntu and I was unable to test 2.6.25.x there due to
Locating the top of the address space ... Address0x0no good?
which AFAIK was a known problem caused by some ubuntu-specific patch.
You can work around this by putting a 0 into /proc/sys/vm/mmap_min_addr.
Quoted 1 line
2.6.26 crashes here with
This one is fixed in 2.6.26.2, as you note below.
Quoted 7 lines
2.6.26.2 has no problems.
Quoted 3 lines
In an earlier email, you provide a nice test case. However, I'm running that here, and getting page after page of successful key generation.
Same here with 2.6.26.2 :(
Hmmm. If it's not a problem there, I'm tempted to call it fixed, whatever it was. Can you check current 2.6.27-rc?
Jeff
Signature · 11 lines
Work email - jdike at linux dot intel dot com ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ User-mode-linux-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Re: [uml-devel] FP registers corruption
Quoted 1 line · Jeff Dike
You can work around this by putting a 0 into
/proc/sys/vm/mmap_min_addr.
Permission denied here - looks quite hard readonly on this kernel even for root.
Quoted 4 lines
Quoted 1 line
Same here with 2.6.26.2 :(
Hmmm. If it's not a problem there, I'm tempted to call it fixed, whatever it was. Can you check current 2.6.27-rc?
OK, I'll try to convince my provider to test 2.6.26.2 and possibly 27-rc soon - I'll let you know of the result.
Thanks
Signature · 11 lines
Stano ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ User-mode-linux-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Re: [uml-devel] FP registers corruption
Quoted 4 lines · Stanislav Meduna
Quoted 1 line
You can work around this by putting a 0 into
/proc/sys/vm/mmap_min_addr.Permission denied here - looks quite hard readonly on this kernel even for root.
Hmmm. The actual fix is below,
Jeff
Signature · 27 lines
Work email - jdike at linux dot intel dot com commit 40fb16a360d9c6459afee91dc793c1e3374feb94 Author: Tom Spink <[email protected]> Date: Thu Jun 5 22:46:12 2008 -0700 uml: deal with inaccessible address space start This patch makes os_get_task_size locate the bottom of the address space, as well as the top. This is for systems which put a lower limit on mmap addresses. It works by manually scanning pages from zero onwards until a valid page is found. Because the bottom of the address space may not be zero, it's not sufficient to assume the top of the address space is the size of the address space. The size is the difference between the top address and bottom address. [[email protected]: changed the name to reflect that this function is supposed to return the top of the process address space, not its size and changed the return value to reflect that. Also some minor formatting changes] Signed-off-by: Tom Spink <[email protected]> Signed-off-by: Jeff Dike <[email protected]> Signed-off-by: Andrew Morton <[email protected]> Signed-off-by: Linus Torvalds <[email protected]>
arch/um/include/os.h +1 -1
@@ -299,6 +299,6 @@ extern int os_arch_prctl(int pid, int code, unsigned long *addr);extern int get_pty (void );/* sys-$ARCH/task_size.c */ -extern unsigned long os_get_task_size (void ); +extern unsigned long os_get_top_address (void );#endif
arch/um/kernel/um_arch.c +1 -1
@@ -274,7 +274,7 @@ int __init linux_main(int argc, char **argv)if (have_root ==0 )add_arg (DEFAULT_COMMAND_LINE ); - host_task_size =os_get_task_size (); + host_task_size =os_get_top_address ();/* * TASK_SIZE needs to be PGDIR_SIZE aligned or else exit_mmap craps * out
arch/um/os-Linux/sys-i386/task_size.c +23 -8
@@ -63,7 +63,7 @@ static int page_ok(unsigned long page)return ok; } -unsigned long os_get_task_size (void ) +unsigned long os_get_top_address (void ) {struct sigaction sa, old;unsigned long bottom =0 ;
@@ -76,9 +76,9 @@ unsigned long os_get_task_size(void)* hosts, but shouldn't hurt otherwise. */ unsigned long top =0xffffd000 >>UM_KERN_PAGE_SHIFT ; -unsigned long test; +unsigned long test, original; -printf ("Locating the top of the address space ... " ); +printf ("Locating the bottom of the address space ... " );fflush (stdout);/*
@@ -89,16 +89,31 @@ unsigned long os_get_task_size(void)sigemptyset (&sa.sa_mask); sa.sa_flags =SA_NODEFER ;if (sigaction (SIGSEGV , &sa, &old)) { -perror ("os_get_task_size" ); +perror ("os_get_top_address" );exit (1 ); } -if (!page_ok (bottom)) { -fprintf (stderr,"Address 0x%x no good?\n" , - bottom <<UM_KERN_PAGE_SHIFT ); +/* Manually scan the address space, bottom-up, until we find +* the first valid page (or run out of them). +*/ +for (bottom =0 ; bottom < top; bottom++) { +if (page_ok (bottom)) +break ; + } + +/* If we've got this far, we ran out of pages. */ +if (bottom == top) { +fprintf (stderr,"Unable to determine bottom of address " +"space.\n" );exit (1 ); } +printf ("0x%x\n" , bottom <<UM_KERN_PAGE_SHIFT ); +printf ("Locating the top of the address space ... " ); +fflush (stdout); + + original = bottom; +/* This could happen with a 4G/4G split */ if (page_ok (top)) goto out;
@@ -114,7 +129,7 @@ unsigned long os_get_task_size(void) out:/* Restore the old SIGSEGV handling */ if (sigaction (SIGSEGV , &old,NULL )) { -perror ("os_get_task_size" ); +perror ("os_get_top_address" );exit (1 ); } top <<=UM_KERN_PAGE_SHIFT ;
arch/um/os-Linux/sys-x86_64/task_size.c +1 -1
@@ -1,4 +1,4 @@ -unsigned long os_get_task_size (unsigned long shift) +unsigned long os_get_top_address (unsigned long shift) {/* The old value of CONFIG_TOP_ADDR */ return 0x7fc0000000 ;
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Re: [uml-devel] FP registers corruption
In reply to Stanislav MedunaQuoted 7 lines · Stanislav Meduna
Quoted 4 lines
Quoted 1 line
Same here with 2.6.26.2 :(
Hmmm. If it's not a problem there, I'm tempted to call it fixed, whatever it was. Can you check current 2.6.27-rc?
OK, I'll try to convince my provider to test 2.6.26.2 and possibly 27-rc soon - I'll let you know of the result.
Sorry for the late response. My provider compiled 2.6.26.3 and the problem is still here :(
I'll try to get the host config from him and to reproduce it in a clean environment with the similar config as he has (2.6.23.17 + skas3, i tried with current and without skas). Is the skas3 patch still maintained?
He doesn't want to switch to SKAS4 yet, as it is marked experimental and requires building from the same tree, which he does not want.
Signature · 11 lines
Stano ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ User-mode-linux-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Re: [uml-devel] FP registers corruption
Quoted 4 lines · Stanislav Meduna
I'll try to get the host config from him and to reproduce it in a clean environment with the similar config as he has (2.6.23.17 + skas3, i tried with current and without skas). Is the skas3 patch still maintained?
An update: I can definitely reproduce the problem using
- 2.6.23.17
- http://user-mode-linux.sourceforge.net/skas3-2.6.23.patch
- a patch from Marek Zelem that adds files missing from your patch (see attach)
Now we don't know whether the files added could cause the problem - they were taken from a version a few weeks older. There was following conversation between you and Marek in June:
Quoted 7 lines
Quoted 4 lines · Marek Zelem
I just want to inform you that skas3 patch available at
http://user-mode-linux.sourceforge.net/skas3-2.6.23.patch
does not contain new files (e.g.proc_mm.c,proc_mm.h, ...).Oops, thanks for letting me know. I'm testing a fixed patch and will update the site when that's done.
Do you have the current version that applies cleanly to the 2.6.23.17? I'd like to test with it before declaring it as a definitive bug, it is well possible that there was some hidden incompatibility.
Regards
Signature · 1 line
Stano
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
Re: [uml-devel] FP registers corruption
Quoted 5 lines · Stanislav Meduna
Quoted 4 lines
I'll try to get the host config from him and to reproduce it in a clean environment with the similar config as he has (2.6.23.17 + skas3, i tried with current and without skas). Is the skas3 patch still maintained?
Quoted 8 lines
An update: I can definitely reproduce the problem using
- 2.6.23.17
- http://user-mode-linux.sourceforge.net/skas3-2.6.23.patch
- a patch from Marek Zelem that adds files missing from your patch
Another piece of information: the FP registers only get corrupted for skas3; running the guest with mode=skas0 or host with CONFIG_PROC_MM disabled works (or at least did not fail in a few minutes - the skas3 fails in seconds).
Regards
Signature · 12 lines
Stano ------------------------------------------------------------------------- This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100&url=/ _______________________________________________ User-mode-linux-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel