Skip to contentSkip to conversation

crypto: Per-CPU cryptd thread implementation based on kcrypto_wq

PATCH In review Huang Ying 3 participants 4 messages Feb 2, 2009 – Feb 4, 2009 Crypto API

lore
Share & tools
lore

Huang Ying

author
#0 lore raw

PATCH 2/3 crypto: Per-CPU cryptd thread implementation based on kcrypto_wq

Original cryptd thread implementation has scalability issue, this patch solve the issue with a per-CPU thread implementation.

struct cryptd_queue is defined to be a per-CPU queue, which holds one struct cryptd_cpu_queue for each CPU. In struct cryptd_cpu_queue, a struct crypto_queue holds all requests for the CPU, a struct work_struct is used to run all requests for the CPU.

Testing based on dm-crypt on an Intel Core 2 E6400 (two cores) machine shows 19.2% performance gain. The testing script is as follow:

-------------------- script begin --------------------------- #!/bin/sh

dmc_create()
{
# Create a crypt device using dmsetup
dmsetup create $2 --table "0 blockdev --getsize $1 crypt cbc(aes-asm)?cryptd?plain:plain babebabebabebabebabebabebabebabe 0 $1 0"
}

dmsetup remove crypt0
dmsetup remove crypt1

dd if=/dev/zero of=/dev/ram0 bs=1M count=4 >& /dev/null dd if=/dev/zero of=/dev/ram1 bs=1M count=4 >& /dev/null

dmc_create /dev/ram0 crypt0
dmc_create /dev/ram1 crypt1

cat >tr.sh <<EOF
#!/bin/sh

for n in \$(seq 10); do

        dd if=/dev/dm-0 of=/dev/null >& /dev/null &
        dd if=/dev/dm-1 of=/dev/null >& /dev/null &

done
wait
EOF

for n in $(seq 10); do
/usr/bin/time sh tr.sh
done
rm tr.sh
-------------------- script end ---------------------------

The separator of dm-crypt parameter is changed from "-" to "?", because "-" is used in some cipher driver name too, and cryptds need to specify cipher driver name instead of cipher name.

The test result on an Intel Core2 E6400 (two cores) is as follow:

without patch:
-----------------wo begin --------------------------
0.04user 0.38system 0:00.39elapsed 107%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6566minor)pagefaults 0swaps
0.07user 0.35system 0:00.35elapsed 121%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6567minor)pagefaults 0swaps
0.06user 0.34system 0:00.30elapsed 135%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6562minor)pagefaults 0swaps
0.05user 0.37system 0:00.36elapsed 119%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6607minor)pagefaults 0swaps
0.06user 0.36system 0:00.35elapsed 120%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6562minor)pagefaults 0swaps
0.05user 0.37system 0:00.31elapsed 136%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6594minor)pagefaults 0swaps
0.04user 0.34system 0:00.30elapsed 126%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6597minor)pagefaults 0swaps
0.06user 0.32system 0:00.31elapsed 125%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6571minor)pagefaults 0swaps
0.06user 0.34system 0:00.31elapsed 134%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6581minor)pagefaults 0swaps
0.05user 0.38system 0:00.31elapsed 138%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6600minor)pagefaults 0swaps
-----------------wo end --------------------------

with patch:
------------------w begin --------------------------
0.02user 0.31system 0:00.24elapsed 141%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6554minor)pagefaults 0swaps
0.05user 0.34system 0:00.31elapsed 127%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6606minor)pagefaults 0swaps
0.07user 0.33system 0:00.26elapsed 155%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6559minor)pagefaults 0swaps
0.07user 0.32system 0:00.26elapsed 151%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6562minor)pagefaults 0swaps
0.05user 0.34system 0:00.26elapsed 150%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6603minor)pagefaults 0swaps
0.03user 0.36system 0:00.31elapsed 124%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6562minor)pagefaults 0swaps
0.04user 0.35system 0:00.26elapsed 147%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6586minor)pagefaults 0swaps
0.03user 0.37system 0:00.27elapsed 146%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6562minor)pagefaults 0swaps
0.04user 0.36system 0:00.26elapsed 154%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6594minor)pagefaults 0swaps
0.04user 0.35system 0:00.26elapsed 154%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+6557minor)pagefaults 0swaps
------------------w end --------------------------

The middle value of elapsed time is:
wo cryptwq: 0.31
w cryptwq: 0.26

The performance gain is about (0.31-0.26)/0.26 = 0.192.

---
 crypto/Kconfig  |    1 
 crypto/cryptd.c |  215 +++++++++++++++++++++++++-------------------------------
 2 files changed, 99 insertions(+), 117 deletions(-)
crypto/Kconfig +1 -0
@@ -114,6 +114,7 @@ config CRYPTO_CRYPTD
 	select CRYPTO_BLKCIPHER
 	select CRYPTO_HASH
 	select CRYPTO_MANAGER
+	select CRYPTO_WORKQUEUE
 	help
 	  This is a generic software asynchronous crypto daemon that
 	  converts an arbitrary synchronous software crypto algorithm
crypto/cryptd.c +98 -117

346 lines not on this page · raw message

#1 lore raw
Quoted 15 lines · On Mon, 02 Feb 2009 14:42:20 +0800 Huang Ying <[email protected]> wrote:

Original cryptd thread implementation has scalability issue, this patch solve the issue with a per-CPU thread implementation.

struct cryptd_queue is defined to be a per-CPU queue, which holds one struct cryptd_cpu_queue for each CPU. In struct cryptd_cpu_queue, a struct crypto_queue holds all requests for the CPU, a struct work_struct is used to run all requests for the CPU.

...

crypto/cryptd.c +5 -0
+int cryptd_init_queue(struct cryptd_queue *queue, unsigned int max_cpu_qlen)
+{
+	int cpu;
+	struct cryptd_cpu_queue *cpu_queue;
+

crypto/cryptd.c struct cryptd_hash_request_ctx { · line 74

Quoted 10 lines
crypto/cryptd.c +10 -0
+	queue->cpu_queue = alloc_percpu(struct cryptd_cpu_queue);
+	if (!queue->cpu_queue)
+		return -ENOMEM;
+	for_each_possible_cpu(cpu) {
+		cpu_queue = per_cpu_ptr(queue->cpu_queue, cpu);
+		crypto_init_queue(&cpu_queue->queue, max_cpu_qlen);
+		INIT_WORK(&cpu_queue->work, cryptd_queue_worker);
+	}
+	return 0;
+}

Could be made static, I believe.

crypto/cryptd.c struct cryptd_hash_request_ctx { · line 86

Quoted 11 lines
crypto/cryptd.c +11 -0
+void cryptd_fini_queue(struct cryptd_queue *queue)
+{
+	int cpu;
+	struct cryptd_cpu_queue *cpu_queue;
+
+	for_each_possible_cpu(cpu) {
+		cpu_queue = per_cpu_ptr(queue->cpu_queue, cpu);
+		BUG_ON(cpu_queue->queue.qlen);
+	}
+	free_percpu(queue->cpu_queue);
+}

Ditto.

Quoted 4 lines
crypto/cryptd.c +4 -0
+int cryptd_enqueue_request(struct cryptd_queue *queue,
+			   struct crypto_async_request *request)
+{
+	int cpu, err;

crypto/cryptd.c struct cryptd_hash_request_ctx { · line 101

Quoted 10 lines
crypto/cryptd.c +10 -0
+	struct cryptd_cpu_queue *cpu_queue;
+
+	cpu = get_cpu();
+	cpu_queue = per_cpu_ptr(queue->cpu_queue, cpu);
+	err = crypto_enqueue_request(&cpu_queue->queue, request);
+	queue_work_on(cpu, kcrypto_wq, &cpu_queue->work);
+	put_cpu();
+
+	return err;
+}

Ditto?

Quoted 1 line

+static void cryptd_queue_worker(struct work_struct *work)

A few comments explaining the code wouldn't kill us...

crypto/cryptd.c struct cryptd_hash_request_ctx { · line 110

Quoted 7 lines
crypto/cryptd.c +7 -0
+{
+	struct cryptd_cpu_queue *cpu_queue;
+	struct crypto_async_request *req, *backlog;
+
+	cpu_queue = container_of(work, struct cryptd_cpu_queue, work);
+	/* Only handle one request at a time to avoid hogging crypto
+	 * workqueue */

Not sure what that means.

crypto/cryptd.c struct cryptd_hash_request_ctx { · line 114

Quoted 4 lines
crypto/cryptd.c +4 -0
+	preempt_disable();
+	backlog = crypto_get_backlog(&cpu_queue->queue);
+	req = crypto_dequeue_request(&cpu_queue->queue);
+	preempt_enable();

This function is run by a per-cpu thread, so it cannot be migrated to another CPU by the CPU scheduler.

It is quite unclear what this preempt_disable() is needed for. Please either remove it or provide adequate comments.

crypto/cryptd.c struct cryptd_hash_request_ctx { · line 124

Quoted 9 lines
crypto/cryptd.c +9 -0
+	if (!req)
+		goto out;
+
+	if (backlog)
+		backlog->complete(backlog, -EINPROGRESS);
+	req->complete(req, 0);
+out:
+	if (cpu_queue->queue.qlen)
+		queue_work(kcrypto_wq, &cpu_queue->work);

Again, unclear and needs commentary.

If we come here via the `goto out;' path above, this CPU queue has no more work do do, I think? If so, why does it requeue itself?

Quoted 5 lines
crypto/cryptd.c +2 -0
+}
+
 
...
maintainer
#2 lore raw
Quoted 10 lines · Andrew Morton

crypto/cryptd.c struct cryptd_hash_request_ctx { · line 110

Quoted 8 lines
crypto/cryptd.c +7 -0
+{
+	struct cryptd_cpu_queue *cpu_queue;
+	struct crypto_async_request *req, *backlog;
+
+	cpu_queue = container_of(work, struct cryptd_cpu_queue, work);
+	/* Only handle one request at a time to avoid hogging crypto
+	 * workqueue */

Not sure what that means.

It looks like this was copied from chainiv which put the work in eventd, which is obviously no longer the case here.

Quoted 12 lines

crypto/cryptd.c struct cryptd_hash_request_ctx { · line 124

Quoted 9 lines
crypto/cryptd.c +9 -0
+	if (!req)
+		goto out;
+
+	if (backlog)
+		backlog->complete(backlog, -EINPROGRESS);
+	req->complete(req, 0);
+out:
+	if (cpu_queue->queue.qlen)
+		queue_work(kcrypto_wq, &cpu_queue->work);

Again, unclear and needs commentary.

Quoted 2 lines

If we come here via the `goto out;' path above, this CPU queue has no more work do do, I think? If so, why does it requeue itself?

This looks like another vestige from chainiv.

Thanks,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
author
#3 lore raw
In reply to Andrew Morton
Quoted 21 lines · Andrew Morton
Quoted 19 lines · On Mon, 02 Feb 2009 14:42:20 +0800 Huang Ying <[email protected]> wrote:

Original cryptd thread implementation has scalability issue, this patch solve the issue with a per-CPU thread implementation.

struct cryptd_queue is defined to be a per-CPU queue, which holds one struct cryptd_cpu_queue for each CPU. In struct cryptd_cpu_queue, a struct crypto_queue holds all requests for the CPU, a struct work_struct is used to run all requests for the CPU.

...

crypto/cryptd.c +9 -0
+int cryptd_init_queue(struct cryptd_queue *queue, unsigned int max_cpu_qlen)
+{
+	int cpu;
+	struct cryptd_cpu_queue *cpu_queue;
+
+	queue->cpu_queue = alloc_percpu(struct cryptd_cpu_queue);
+	if (!queue->cpu_queue)
+		return -ENOMEM;
+	for_each_possible_cpu(cpu) {
Quoted 8 lines

crypto/cryptd.c struct cryptd_hash_request_ctx { · line 74

Quoted 6 lines
crypto/cryptd.c +6 -0
+		cpu_queue = per_cpu_ptr(queue->cpu_queue, cpu);
+		crypto_init_queue(&cpu_queue->queue, max_cpu_qlen);
+		INIT_WORK(&cpu_queue->work, cryptd_queue_worker);
+	}
+	return 0;
+}

Could be made static, I believe.

OK. I will make it static.

Quoted 5 lines
Quoted 5 lines
crypto/cryptd.c +5 -0
+void cryptd_fini_queue(struct cryptd_queue *queue)
+{
+	int cpu;
+	struct cryptd_cpu_queue *cpu_queue;
+
Quoted 8 lines

crypto/cryptd.c struct cryptd_hash_request_ctx { · line 86

Quoted 6 lines
crypto/cryptd.c +6 -0
+	for_each_possible_cpu(cpu) {
+		cpu_queue = per_cpu_ptr(queue->cpu_queue, cpu);
+		BUG_ON(cpu_queue->queue.qlen);
+	}
+	free_percpu(queue->cpu_queue);
+}

Ditto.

As above.

Quoted 8 lines
Quoted 8 lines
crypto/cryptd.c +8 -0
+int cryptd_enqueue_request(struct cryptd_queue *queue,
+			   struct crypto_async_request *request)
+{
+	int cpu, err;
+	struct cryptd_cpu_queue *cpu_queue;
+
+	cpu = get_cpu();
+	cpu_queue = per_cpu_ptr(queue->cpu_queue, cpu);
Quoted 8 lines

crypto/cryptd.c struct cryptd_hash_request_ctx { · line 101

Quoted 6 lines
crypto/cryptd.c +6 -0
+	err = crypto_enqueue_request(&cpu_queue->queue, request);
+	queue_work_on(cpu, kcrypto_wq, &cpu_queue->work);
+	put_cpu();
+
+	return err;
+}

Ditto?

As above.

Quoted 3 lines
Quoted 1 line

+static void cryptd_queue_worker(struct work_struct *work)

A few comments explaining the code wouldn't kill us...

OK. I will add some comments. This the function for work_struct in cryptd_cpu_queue, it does real encryption/decryption.

Quoted 9 lines

crypto/cryptd.c struct cryptd_hash_request_ctx { · line 110

Quoted 7 lines
crypto/cryptd.c +7 -0
+{
+	struct cryptd_cpu_queue *cpu_queue;
+	struct crypto_async_request *req, *backlog;
+
+	cpu_queue = container_of(work, struct cryptd_cpu_queue, work);
+	/* Only handle one request at a time to avoid hogging crypto
+	 * workqueue */

Not sure what that means.

cryptd is not the only user of kcrypto_wq (now, chainiv uses it too), and there may be a lot of requests in cryptd_cpu_queue, to reduce the latency of other kcrypto_wq user, only one request is processed in work_struct function, then the work_struct will be inserted again.

Quoted 10 lines

crypto/cryptd.c struct cryptd_hash_request_ctx { · line 114

Quoted 4 lines
crypto/cryptd.c +4 -0
+	preempt_disable();
+	backlog = crypto_get_backlog(&cpu_queue->queue);
+	req = crypto_dequeue_request(&cpu_queue->queue);
+	preempt_enable();

This function is run by a per-cpu thread, so it cannot be migrated to another CPU by the CPU scheduler.

It is quite unclear what this preempt_disable() is needed for. Please either remove it or provide adequate comments.

preempt_disable is used to protect cpu_queue->queue. It can be accessed by cryptd_enqueue_request(). When cryptd_queue_worker() is running, it can be preempted by cryptd_enqueue_request() on same CPU.

Quoted 12 lines

crypto/cryptd.c struct cryptd_hash_request_ctx { · line 124

Quoted 9 lines
crypto/cryptd.c +9 -0
+	if (!req)
+		goto out;
+
+	if (backlog)
+		backlog->complete(backlog, -EINPROGRESS);
+	req->complete(req, 0);
+out:
+	if (cpu_queue->queue.qlen)
+		queue_work(kcrypto_wq, &cpu_queue->work);

Again, unclear and needs commentary.

Quoted 2 lines

If we come here via the `goto out;' path above, this CPU queue has no more work do do, I think? If so, why does it requeue itself?

If there is no more work to do, the cpu_queue->queue.qlen will be zero, so it will not request itself. If we come here via another path, we may need do that.

Best Regards,
Huang Ying