Skip to contentSkip to conversation

Couldn't obtain random bytes in sshd - problem in RAND_poll?

uml-devel Stanislav Meduna 4 participants 5 messages Aug 6, 2008 – Aug 7, 2008 User-mode Linux (UML)

lore
Share & tools
lore

Stanislav Meduna

author
#0 lore raw

Hi,

I and a few other users are seeing sshd failing with
Couldn't obtain random bytes (error 604389476)
and other ssl-related application failing randomly in user mode linux guests and I suspect a problem in openssl that got triggered by some change in UML.

I reviewed the RAND_poll function in rand_unix.c (statically, no time for building a debug version now) and have following suspicions:

===
For Linux:

int r; ... this has random bytes from stack
...

if (poll(&pset, 1, usec / 1000) < 0)
usec = 0;
else
try_read = (pset.revents & POLLIN) != 0;

... Let's say that the poll timed out (i.e. returned 0)
try_read remains 0, r still has garbage

while ((r > 0 || (errno == EINTR || errno == EAGAIN)) &&

... Let's say that the garbage was negative. We are out of
the loop and errno has bogus data (successfull/timed out poll did not set anything)

=== For other Unices there's additional problem:

If the select select's successfully and immediately, it can leave the time not slept unchanged in the time argument (which is IMHO fully legal, if it finds the bytes immediately). If the read then does not get all the needed bytes, the code
if (usec == 10*1000)

     usec = 0;

kicks in and we are out of the loop again.

Suggested changes:

- add

     r = -1;

inside the do loop after the int try_read = 0;

- change

     if (usec == 10*1000)
   into
     if (r < 0 && usec == 10*1000)

Regards

Signature · 11 lines
                                     Stano

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
#1 lore raw
Quoted 18 lines · Stanislav Meduna

Hi,

I and a few other users are seeing sshd failing with
Couldn't obtain random bytes (error 604389476)
and other ssl-related application failing randomly in user mode linux guests and I suspect a problem in openssl that got triggered by some change in UML.

I reviewed the RAND_poll function in rand_unix.c (statically, no time for building a debug version now) and have following suspicions:

===
For Linux:

int r; ... this has random bytes from stack
...

Quoted 7 lines

if (poll(&pset, 1, usec / 1000) < 0)
usec = 0;
else
try_read = (pset.revents & POLLIN) != 0;

... Let's say that the poll timed out (i.e. returned 0)
try_read remains 0, r still has garbage

The r is set to -1 in the else of if(try_read) statement.

Quoted 5 lines

while ((r > 0 || (errno == EINTR || errno == EAGAIN)) &&

... Let's say that the garbage was negative. We are out of
the loop and errno has bogus data (successfull/timed out poll did not set anything)

errno has garbage value - this should be fixed by initializing errno to 0 before the poll/select calls. But in the worst case it means the code will wait until it will be able to read some data from the random device (which was not the intent).

Quoted 10 lines

=== For other Unices there's additional problem:

If the select select's successfully and immediately, it can leave the time not slept unchanged in the time argument (which is IMHO fully legal, if it finds the bytes immediately). If the read then does not get all the needed bytes, the code
if (usec == 10*1000)

     usec = 0;

kicks in and we are out of the loop again.

The problem is not in the RAND_poll() timeouting - this is fully intentional, the function should timeout after 10ms if the random device blocks read. The default is to
try /dev/urandom, /dev/random, /dev/srandom in this order. So if you for example do not have /dev/urandom and have just the blocking /dev/random, it is perfectly possible that the RAND_poll returns error. The other possibility is that the /dev/urandom is broken in UML and blocks if not enough entropy is available.

Signature · 14 lines
Tomas Mraz
No matter how far down the wrong road you've gone, turn back.
                                              Turkish proverb


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
author
#2 lore raw
Quoted 2 lines · Tomas Mraz

errno has garbage value - this should be fixed by initializing errno to 0 before the poll/select calls.

Actually after it returns with timeout - a successfull syscall is free to set errno to whatever value it wants, it is only after an error the value has to be meaningful (I did have this problem a few times).

Quoted 3 lines

The problem is not in the RAND_poll() timeouting - this is fully intentional, the function should timeout after 10ms if the random device blocks read.

Ah, ok..

So what should the applications calling openssl actually do if this happens? Now the ssh/apache/... simply exit, which is bad (it left me without an access to a remote box...).

I assume they are not calling the method directly, instead they are using some of the openssl's methods. In the current situation anyone who actually wants to block until the entropy is available is simply out of luck :(

Quoted 3 lines

try /dev/urandom, /dev/random, /dev/srandom in this order. So if you for example do not have /dev/urandom and have just the blocking /dev/random, it is perfectly possible that the RAND_poll returns error.

Both UML guest and host have /dev/urandom. I straced a ssh, it opens /dev/urandom first, so this should be OK too.

Quoted 2 lines

The other possibility is that the /dev/urandom is broken in UML and blocks if not enough entropy is available.

Good.. let's try it:

===
#include <unistd.h>
#include <fcntl.h>
#include <poll.h>
#include <stdio.h>

main()
{
int fd = open("/dev/urandom", O_RDONLY|O_NONBLOCK|O_NOCTTY); int i;
int errpoll=0, blocked=0, rdbytes=0,errread=0, nullread=0;

   for (i=0; i < 1000000; ++i)
   {
     struct pollfd pset;
     int r;
     char tmp[32];
     pset.fd = fd;
     pset.events = POLLIN;
     pset.revents = 0;

     r = poll(&pset, 1, 10);
     if (r > 0)
     {
       if ((pset.revents & POLLIN) != 0)
       {
         r = read(fd, tmp, sizeof(tmp));
         if (r < 0)
           errread++;
         else if (r==0)
           nullread++;
         else
           rdbytes += r;
       }
       else
       {
         printf("poll returned %d, but POLLIN is false (%x)\n", r, pset.revents);
       }
     }
     else if (r == 0)
       blocked++;
     else
       errpoll++;

}

printf("got %d bytes of entropy, poll err %d, blocked %d times, err read: %d, null
read: %d\n", rdbytes, errpoll, blocked, errread, nullread);
}
===

got 3200000 bytes of entropy, poll err 0, blocked 0 times, err read: 0, null read: 0

Tried many many times, even two running at the same time or poll timeout set to zero, not one instance of blocking even with
od -x /dev/urandom
and
od -x /dev/random
running simultaneously (the second one blocks, of course).

Hmmmm.. what the #$%# is happening here.. more ideas?

Signature · 11 lines
                                    Stano

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel
#3 lore raw
Quoted 11 lines

Tried many many times, even two running at the same time or poll timeout set to zero, not one instance of blocking even with
od -x /dev/urandom
and
od -x /dev/random
running simultaneously (the second one blocks, of course).

Quoted 4 lines

Hmmmm.. what the #$%# is happening here.. more ideas?

--
Stano

My bet is that '/dev/urandom' only blocks if it doesn't have enough entropy. Early in the startup process, '/dev/urandom' doesn't have enough entropy, and your application times out on it.

Later on, when the system has had lots of network activity, you log in and test '/dev/urandom'. At this point, the system is well-seeded from the network activity. So it works great for you.

Try launching your test program automatically on boot up at the saem time you launch ssh or whatever application is failing. I bet '/dev/urandom' will fail then.

If you have a network, one solution might be to do a few 'ping's or 'nslookup's to seed the entropy pool. You can also keep an entropy pool on disk, saving it on shutdown and loading it on startup.

Or I could be completely wrong.

DS

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

#4 lore raw
In reply to Stanislav Meduna
Quoted 4 lines · Stanislav Meduna

So what should the applications calling openssl actually do if this happens? Now the ssh/apache/... simply exit, which is bad (it left me without an access to a remote box...).

Exiting is the best behaviour - continuing without a good source of randomness may compromise cryptographic protocols and even long-term private keys (e.g. if DSA is used).

-d

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK & win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
User-mode-linux-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel