Skip to contentSkip to conversation

lustre: why does cfs_get_random_bytes() exist?

DISCUSSION Theodore Ts'o 4 participants 8 messages Oct 3, 2013 – Oct 5, 2013

lore
Share & tools
lore
#0 lore raw

I've been auditing uses of get_random_bytes() since there are places where get_random_bytes() is getting used where something weaker, such as prandom_u32() is quite sufficient. Basically, if kernel code just needs a random number which does not have any cryptographic requirements (such as in ext[234]. which gets the new block group used for inode allocations using get_random_bytes), then prandom_u32() should be used instead of get_random_bytes() to save CPU overhead and to reduce the drain on the /dev/urandom's entropy pool.

Typically, the reason for this is either for historical reasons, since prandom_u32() hadn't existed when the code was written, or because historical code was cut and pasted into newer code.

When I came across staging/lustre/lustre/libcfs/prng.c, I saw something which is **really** weird. It defines a cfs_rand() which is functionally identical to prandom_u32(). More puzzlingly, it also defines cfs_get_random_bytes() which calls get_random_bytes() and then xor's the result with cfs_rand(). That last step has no cryptographic effect, so I'm really wondering who thought this as a good idea and/or necessary.

What I think should happen is that staging/lustre/lustre/libcfs/prng.c should be removed, and calls to cfs_rand() should get replaced prandom_u32(), and cfs_get_random_bytes() should get replaced with get_random_bytes().

Does this sound reasonable?

Cheers,

- Ted

#1 lore raw
Quoted 21 lines · Theodore Ts'o

I've been auditing uses of get_random_bytes() since there are places where get_random_bytes() is getting used where something weaker, such as prandom_u32() is quite sufficient. Basically, if kernel code just needs a random number which does not have any cryptographic requirements (such as in ext[234]. which gets the new block group used for inode allocations using get_random_bytes), then prandom_u32() should be used instead of get_random_bytes() to save CPU overhead and to reduce the drain on the /dev/urandom's entropy pool.

Typically, the reason for this is either for historical reasons, since prandom_u32() hadn't existed when the code was written, or because historical code was cut and pasted into newer code.

When I came across staging/lustre/lustre/libcfs/prng.c, I saw something which is **really** weird. It defines a cfs_rand() which is functionally identical to prandom_u32(). More puzzlingly, it also defines cfs_get_random_bytes() which calls get_random_bytes() and then xor's the result with cfs_rand(). That last step has no cryptographic effect, so I'm really wondering who thought this as a good idea and/or necessary.

Quoted 6 lines

What I think should happen is that staging/lustre/lustre/libcfs/prng.c should be removed, and calls to cfs_rand() should get replaced prandom_u32(), and cfs_get_random_bytes() should get replaced with get_random_bytes().

Does this sound reasonable?

Sounds reasonable to me, care to send a patch to do so?

thanks,

greg k-h

#2 lore raw
Quoted 4 lines · Greg KH
Quoted 1 line

Does this sound reasonable?

Sounds reasonable to me, care to send a patch to do so?

I can do that, but I was waiting for Andras, Peng or Nikita to let me now if there was something I was missing or not. I'm pretty sure it's something bogus, perhaps left over from a OS abstraction layer to support Solaris or some such, but I am curious what was the historical reason for the current code.

Cheers,

- Ted

#3 lore raw
Quoted 11 lines · Theodore Ts'o
Quoted 4 lines · Greg KH
Quoted 1 line

Does this sound reasonable?

Sounds reasonable to me, care to send a patch to do so?

I can do that, but I was waiting for Andras, Peng or Nikita to let me now if there was something I was missing or not. I'm pretty sure it's something bogus, perhaps left over from a OS abstraction layer to support Solaris or some such, but I am curious what was the historical reason for the current code.

The root of the problem with the existing get_random_bytes() is that when the
Cray supercomputers with around 20k nodes reboot all at the same instant, we
had problems with the Lustre client UUID (originally from generate_random_uuid())
being identical between multiple nodes since they have virtually no entropy at
boot time, and they and did not have hardware RNG.

The Lustre cfs_get_random_bytes() incorporates (via cfs_rand()) a seed which
also hashes in the addresses from any network interfaces that are configured.
Conversely, cfs_rand() also is seeded at startup from get_random_bytes() in case a hardware RNG is available. This ensures even with identical initial conditions cfs_get_random_bytes() gets a different random stream on each node.

It is also true that cfs_get_random_bytes() and cfs_rand() predate the newer
APIs such as prandom_u32() (which, to be honest, I didn't even know existed today). This is work that was done for maybe 2.4.29? (RH 9?) kernels.

The number of users of cfs_get_random_bytes() is relatively small, mostly just
for the client UUID and other unique identifiers at startup time. The only place it is used on an ongoing basis is for the capabilities that are passed
from the metadata server to clients so the clients can validate they have access
to the objects on the data server. The rest of the code uses cfs_rand() for
values that only need to be statistically uniform over time, so they don't deplete the entropy pool.

I'm not against cleaning this up, if there is some mechanism for the startup
code to add in the node interface addresses into the entropy pool, and this is also used to perturb the prandom_u32() sequence after that point. Besides
that initialization, it would be a simple search & replace for the users.

Cheers, Andreas

Signature · 4 lines
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division
#4 lore raw
Quoted 9 lines · Andreas

The Lustre cfs_get_random_bytes() incorporates (via cfs_rand()) a seed which
also hashes in the addresses from any network interfaces that are configured.
Conversely, cfs_rand() also is seeded at startup from get_random_bytes() in case a hardware RNG is available. This ensures even with identical initial conditions cfs_get_random_bytes() gets a different random stream on each node.

With modern kernels, the /dev/random driver has the add_device_randomness() interface which is used to mix in personalization information, which includes the network MAC address. So that particular concern should be covered without the hack of mixing in cfs_rand().

Quoted 4 lines

I'm not against cleaning this up, if there is some mechanism for the startup code to add in the node interface addresses into the entropy pool, and this is also used to perturb the prandom_u32() sequence after that point.

That's handled too, via the late initcall prandom_reseed().

Cheers,

- Ted

#5 lore raw
Quoted 12 lines · Theodore Ts'o
Quoted 10 lines · Andreas

The Lustre cfs_get_random_bytes() incorporates (via cfs_rand()) a seed which also hashes in the addresses from any network interfaces that are configured.
Conversely, cfs_rand() also is seeded at startup from get_random_bytes() in
case a hardware RNG is available. This ensures even with identical initial
conditions cfs_get_random_bytes() gets a different random stream on each node.

Quoted 5 lines

With modern kernels, the /dev/random driver has the add_device_randomness() interface which is used to mix in personalization information, which includes the network MAC address. So that particular concern should be covered without the hack of mixing in cfs_rand().

I think that depends on the network driver. The Cray systems have some very strange networking hardware that is beyond our control - definitely not ethernet or Infiniband.

I'll have to ask the Cray folks if their network drivers do this today.

Quoted 11 lines
Quoted 4 lines

I'm not against cleaning this up, if there is some mechanism for the startup code to add in the node interface addresses into the entropy pool, and this is also used to perturb the prandom_u32() sequence after that point.

That's handled too, via the late initcall prandom_reseed().

Cheers,

- Ted

Cheers, Andreas

Signature · 4 lines
Andreas Dilger

Lustre Software Architect
Intel High Performance Data Division
#6 lore raw
Quoted 9 lines · Andreas
Quoted 5 lines

With modern kernels, the /dev/random driver has the add_device_randomness() interface which is used to mix in personalization information, which includes the network MAC address. So that particular concern should be covered without the hack of mixing in cfs_rand().

I think that depends on the network driver. The Cray systems have some very strange networking hardware that is beyond our control - definitely not ethernet or Infiniband.

add_device_randomness() is called from __dev_open() and dev_set_mac_address() in net/core/dev.c. This is above the ethernet and infiniband level. So as long as it looks like a Linux network device, and they are setting the hardware media access address in the standard place (dev->dev_addr), it should work fine.

If they don't then they should fix their drivers to call add_device_randomness(); the answer shouldn't be to make every single users of the Linux random number generation infrastructure work around the problem at the subsystem or file system level!

- Ted

#7 lore raw
Quoted 10 lines · Theodore Ts'o

add_device_randomness() is called from __dev_open() and dev_set_mac_address() in net/core/dev.c. This is above the ethernet and infiniband level. So as long as it looks like a Linux network device, and they are setting the hardware media access address in the standard place (dev->dev_addr), it should work fine.

If they don't then they should fix their drivers to call add_device_randomness(); the answer shouldn't be to make every single users of the Linux random number generation infrastructure work around the problem at the subsystem or file system level!

Besides these points a filesystem isn't the place to work around entropy pool issues on some obscure platform. Even if they can't fix it this way for some reason the workaround still should be in the interaction of the arch code and the Linux prng infrastructure and not in Lustre.