Skip to contentSkip to conversation

pdflush and dm-crypt

DISCUSSION Zoltan NAGY 5 participants 8 messages Mar 29, 2004 – Mar 30, 2004

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi!

I've just upgraded the system to 2.6.5-rc2-bk6, and I'm using dm-crypt. It's a heavily used server, on average 20-30mbit/sec traffic is on the wire 7/24, and just noticed, that the load is very high. In every 4-5 sec pdflush takes a lot of cpu... Is this intentional? I've found a similar question on kerneltrap
(http://kerneltrap.org/node/view/2756), but havent found a solution yet. I'm just wondering if it is a problem, or it's the normal behavior? It's a 1.8 P4 with 1G ram and highmem enabled, with 256 bit aes thru dm-crypt.

Regards,

--
Zoltan NAGY,
Network Administrator

#1 lore raw
Quoted 11 lines · Zoltan NAGY

Hi!

I've just upgraded the system to 2.6.5-rc2-bk6, and I'm using dm-crypt. It's a heavily used server, on average 20-30mbit/sec traffic is on the wire 7/24, and just noticed, that the load is very high. In every 4-5 sec pdflush takes a lot of cpu... Is this intentional? I've found a similar question on kerneltrap
(http://kerneltrap.org/node/view/2756), but havent found a solution yet. I'm just wondering if it is a problem, or it's the normal behavior? It's a 1.8 P4 with 1G ram and highmem enabled, with 256 bit aes thru dm-crypt.

Try the -mm kernels intead, should have lots better behaviour for pdflush/dm interactions.

Signature · 1 line
Jens Axboe
#2 lore raw
Quoted 14 lines · Jens Axboe
Quoted 11 lines · Zoltan NAGY

Hi!

I've just upgraded the system to 2.6.5-rc2-bk6, and I'm using dm-crypt. It's a heavily used server, on average 20-30mbit/sec traffic is on the wire 7/24, and just noticed, that the load is very high. In every 4-5 sec pdflush takes a lot of cpu... Is this intentional? I've found a similar question on kerneltrap
(http://kerneltrap.org/node/view/2756), but havent found a solution yet. I'm just wondering if it is a problem, or it's the normal behavior? It's a 1.8 P4 with 1G ram and highmem enabled, with 256 bit aes thru dm-crypt.

Quoted 3 lines

Try the -mm kernels intead, should have lots better behaviour for pdflush/dm interactions.

How come? Isn't this problem just "gee, we have a lot of stuff to encrypt during writeback"? If so, then it should be sufficient to poke a hole in the encryption loop?

drivers/md/dm-crypt.c renamed from drivers/md/dm-crypt.c~a +1 -0
@@ -669,6 +669,7 @@ static int crypt_map(struct dm_target *t
 		/* out of memory -> run queues */
 		if (remaining)
 			blk_congestion_wait(bio_data_dir(clone), HZ/100);
+		cond_resched();
 	}
 
 	/* drop reference, clones could have returned before we reach this */

_

#3 lore raw
Quoted 15 lines · Andrew Morton
Quoted 14 lines · Jens Axboe
Quoted 11 lines · Zoltan NAGY

Hi!

I've just upgraded the system to 2.6.5-rc2-bk6, and I'm using dm-crypt. It's a heavily used server, on average 20-30mbit/sec traffic is on the wire 7/24, and just noticed, that the load is very high. In every 4-5 sec pdflush takes a lot of cpu... Is this intentional? I've found a similar question on kerneltrap
(http://kerneltrap.org/node/view/2756), but havent found a solution yet. I'm just wondering if it is a problem, or it's the normal behavior? It's a 1.8 P4 with 1G ram and highmem enabled, with 256 bit aes thru dm-crypt.

Quoted 7 lines
Quoted 3 lines

Try the -mm kernels intead, should have lots better behaviour for pdflush/dm interactions.

How come? Isn't this problem just "gee, we have a lot of stuff to encrypt during writeback"? If so, then it should be sufficient to poke a hole in the encryption loop?

If that is the problem, then yeah that'd work. I was assuming the 'load' was just io load and pdflush got stuck on them.

drivers/md/dm-crypt.c static int crypt_map(struct dm_target *t · line 673

Quoted 8 lines
drivers/md/dm-crypt.c renamed from drivers/md/dm-crypt.c~a +1 -0
@@ -669,6 +669,7 @@ static int crypt_map(struct dm_target *t
 		/* out of memory -> run queues */
 		if (remaining)
 			blk_congestion_wait(bio_data_dir(clone), HZ/100);
+		cond_resched();
 	}

Reminds me, we have to kill that blk_congestion_wait() stuff too.

Signature · 1 line
Jens Axboe
#4 lore raw
In reply to Andrew Morton

Am Mo, den 29.03.2004, um 16:12 Uhr -0800, schrieb Andrew Morton:

Quoted 7 lines

How come? Isn't this problem just "gee, we have a lot of stuff to encrypt during writeback"? If so, then it should be sufficient to poke a hole in the encryption loop?

drivers/md/dm-crypt.c static int crypt_map(struct dm_target *t · line 675

Quoted 7 lines
drivers/md/dm-crypt.c renamed from drivers/md/dm-crypt.c~a +1 -0
@@ -669,6 +669,7 @@ static int crypt_map(struct dm_target *t
 		/* out of memory -> run queues */
 		if (remaining)
 			blk_congestion_wait(bio_data_dir(clone), HZ/100);
+		cond_resched();
 	}
 
 	/* drop reference, clones could have returned before we reach this */

cryptoapi always does this after every block. It also happens with preemption enabled. I got feedback from a person who said that renicing pdflush to 0 helped. So it looks like the CPU scheduler doesn't want to schedule pdflush away. Hmm.

#5 lore raw
Quoted 16 lines · Christophe Saout

Am Mo, den 29.03.2004, um 16:12 Uhr -0800, schrieb Andrew Morton:

Quoted 13 lines

How come? Isn't this problem just "gee, we have a lot of stuff to encrypt during writeback"? If so, then it should be sufficient to poke a hole in the encryption loop?

drivers/md/dm-crypt.c renamed from drivers/md/dm-crypt.c~a +1 -0
@@ -669,6 +669,7 @@ static int crypt_map(struct dm_target *t
 		/* out of memory -> run queues */
 		if (remaining)
 			blk_congestion_wait(bio_data_dir(clone), HZ/100);
+		cond_resched();
 	}
 
Quoted 6 lines

drivers/md/dm-crypt.c static int crypt_map(struct dm_target *t · line 675

Quoted 1 line
drivers/md/dm-crypt.c renamed from drivers/md/dm-crypt.c~a +0 -0
 	/* drop reference, clones could have returned before we reach this */

cryptoapi always does this after every block. It also happens with preemption enabled. I got feedback from a person who said that renicing pdflush to 0 helped. So it looks like the CPU scheduler doesn't want to schedule pdflush away. Hmm.

Oh, OK, since pdflush was converted to use the kthread stuff it has been running at keventd's `nice -10'. You can probably renice it by hand.

mm/pdflush.c renamed from mm/pdflush.c~pdflush-nice-0 +6 -0
diff -puN mm/pdflush.c~pdflush-nice-0 mm/pdflush.c
@@ -177,6 +177,12 @@ static int __pdflush(struct pdflush_work
 static int pdflush(void *dummy)
 {
 	struct pdflush_work my_work;
+
+	/*
+	 * pdflush can spend a lot of time doing encryption via dm-crypt.  We
+	 * don't want to do that at keventd's priority.
+	 */
+	set_user_nice(current, 0);
 	return __pdflush(&my_work);
 }
 

_

#6 lore raw
In reply to Jens Axboe

Hello Jens,

Quoted 2 lines

Try the -mm kernels intead, should have lots better behaviour for pdflush/dm interactions.

allright. with -mm5 it is _much_ better. load is back to normal. I suppose we should get all this stuff into mainline ASAP ;) I'm sure lot of people are using this combo.

thanks for you help,

--
Zoltan NAGY,
Network Administrator

#7 lore raw
In reply to Zoltan NAGY
Quoted 9 lines · Zoltan NAGY

I've just upgraded the system to 2.6.5-rc2-bk6, and I'm using dm-crypt. It's a heavily used server, on average 20-30mbit/sec traffic is on the wire 7/24, and just noticed, that the load is very high. In every 4-5 sec pdflush takes a lot of cpu... Is this intentional? I've found a similar question on kerneltrap
(http://kerneltrap.org/node/view/2756), but havent found a solution yet. I'm just wondering if it is a problem, or it's the normal behavior? It's a 1.8 P4 with 1G ram and highmem enabled, with 256 bit aes thru dm-crypt.

Same here (duron 1.2GHz, 512MB ram, IDE disks (via and promise controllers)) and it doesn't matter if I'm using cryptoloop or dm-crypt.

System hangs for about one seconds (xmms stops playing, xterms are not responding to key presses) and then everything back to normal until next hang happens (in few seconds).

Quoted 2 lines

Zoltan NAGY,
Network Administrator

ps. I'm not sure (didn't do any checking to be sure) but afaik this started to happening with 2.6.3 kernel.

Signature · 2 lines
Arkadiusz Miśkiewicz     CS at FoE, Wroclaw University of Technology
arekm.pld-linux.org, 1024/3DB19BBD, JID: arekm.jabber.org, PLD/Linux