multiple crypt devices not using multiple CPUs

Hi,

I guess I must be missing something. I had understood that if one had multiple block devices being crypted by dm-crypt, if they were being operated on at the same time, their parallel crypto operations would utilize different CPUs.

So I just did the following:

# lvcreate -L10G -n t1 vg
# cryptsetup --verbose --verify-passphrase luksFormat /dev/vg/t1
# cryptsetup luksOpen /dev/vg/t1 t1-use
# dd of=/dev/null if=/dev/mapper/t1-use bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 12.1207 s, 86.5 MB/s

observe (a series of) "kworker(s)" each using near 100% of a core to carry out the crypto. I say a series of kworkers since it appears to almost be a "relay" of them, each handing the work of to another and going away. But all in all there is only every near 1 full core's worth running.

So now...

# lvcreate -L10G -n t2 vg
# cryptsetup --verbose --verify-passphrase luksFormat /dev/vg/t2
# cryptsetup luksOpen /dev/vg/t2 t2-use
# for ((x=1; x<3; x++)); do

[1] 7369
[2] 7370
# 1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 27.62 s, 38.0 MB/s 1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 27.8015 s, 37.7 MB/s

Here I observe two kworker threads each near 50% of a core. Clearly they are both using the same core. But why? As as an aside, aggregate throughput of the two threads is quite a bit less than a single thread.

But what I really want to understand is why crypto kworkers are not being scheduled to different, idle cores.

Cheers,
b.

In current kernels, work keeps on cpu which submitted it.

Please read
http://www.saout.de/pipermail/dm-crypt/2012-July/002582.html

(and try to run the second case with direct-io, iow add oflag=direct for dd)

m.

So a single process doing parallel reads from multiple devices (I'm sure you could guess where I'm going with this) will never see more throughput than it would from a single device? If so, all the more reason to get multi-core support (for a single process of course) involved. :-/

Yeah. I'm trying to see this "more processes" behavior but even the two dds each running in different shells are still attacking the same core.

oflag=direct?

# dd oflag=direct of=/dev/null if=/dev/mapper/t1-use bs=1M count=1000
dd: opening ‘/dev/null’: Invalid argument

I guess O_DIRECT it not valid for /dev/null.

Cheers,
b.