pdflush and dm-crypt
Subsystems (2)
Hi!
I've just upgraded the system to 2.6.5-rc2-bk6, and I'm using dm-crypt. It's a heavily used server, on average 20-30mbit/sec traffic is on the wire 7/24, and just noticed, that the load is very high. In every 4-5 sec pdflush takes a lot of cpu... Is this intentional? I've found a similar question on kerneltrap
(http://kerneltrap.org/node/view/2756), but havent found a solution yet. I'm just wondering if it is a problem, or it's the normal behavior? It's a 1.8 P4 with 1G ram and highmem enabled, with 256 bit aes thru dm-crypt.
Regards,
--
Zoltan NAGY,
Network Administrator
Quoted 11 lines · Zoltan NAGY
Hi!
I've just upgraded the system to 2.6.5-rc2-bk6, and I'm using dm-crypt. It's a heavily used server, on average 20-30mbit/sec traffic is on the wire 7/24, and just noticed, that the load is very high. In every 4-5 sec pdflush takes a lot of cpu... Is this intentional? I've found a similar question on kerneltrap
(http://kerneltrap.org/node/view/2756), but havent found a solution yet. I'm just wondering if it is a problem, or it's the normal behavior? It's a 1.8 P4 with 1G ram and highmem enabled, with 256 bit aes thru dm-crypt.
Try the -mm kernels intead, should have lots better behaviour for pdflush/dm interactions.
Signature · 1 line
Jens Axboe
Quoted 14 lines · Jens Axboe
Quoted 11 lines · Zoltan NAGY
Hi!
I've just upgraded the system to 2.6.5-rc2-bk6, and I'm using dm-crypt. It's a heavily used server, on average 20-30mbit/sec traffic is on the wire 7/24, and just noticed, that the load is very high. In every 4-5 sec pdflush takes a lot of cpu... Is this intentional? I've found a similar question on kerneltrap
(http://kerneltrap.org/node/view/2756), but havent found a solution yet. I'm just wondering if it is a problem, or it's the normal behavior? It's a 1.8 P4 with 1G ram and highmem enabled, with 256 bit aes thru dm-crypt.
Quoted 3 lines
Try the -mm kernels intead, should have lots better behaviour for pdflush/dm interactions.
How come? Isn't this problem just "gee, we have a lot of stuff to encrypt during writeback"? If so, then it should be sufficient to poke a hole in the encryption loop?
drivers/md/dm-crypt.c renamed from drivers/md/dm-crypt.c~a +1 -0
@@ -669,6 +669,7 @@ static int crypt_map(struct dm_target *t/* out of memory -> run queues */ if (remaining)blk_congestion_wait (bio_data_dir (clone),HZ /100 ); +cond_resched (); }/* drop reference, clones could have returned before we reach this */
_
Quoted 15 lines · Andrew Morton
Quoted 14 lines · Jens Axboe
Quoted 11 lines · Zoltan NAGY
Hi!
I've just upgraded the system to 2.6.5-rc2-bk6, and I'm using dm-crypt. It's a heavily used server, on average 20-30mbit/sec traffic is on the wire 7/24, and just noticed, that the load is very high. In every 4-5 sec pdflush takes a lot of cpu... Is this intentional? I've found a similar question on kerneltrap
(http://kerneltrap.org/node/view/2756), but havent found a solution yet. I'm just wondering if it is a problem, or it's the normal behavior? It's a 1.8 P4 with 1G ram and highmem enabled, with 256 bit aes thru dm-crypt.
Quoted 7 lines
Quoted 3 lines
Try the -mm kernels intead, should have lots better behaviour for pdflush/dm interactions.
How come? Isn't this problem just "gee, we have a lot of stuff to encrypt during writeback"? If so, then it should be sufficient to poke a hole in the encryption loop?
If that is the problem, then yeah that'd work. I was assuming the 'load' was just io load and pdflush got stuck on them.
drivers/md/dm-crypt.c static int crypt_map(struct dm_target *t · line 673
Quoted 8 lines
drivers/md/dm-crypt.c renamed from drivers/md/dm-crypt.c~a +1
-0@@ -669,6 +669,7 @@ static int crypt_map(struct dm_target *t/* out of memory -> run queues */ if (remaining)blk_congestion_wait (bio_data_dir (clone),HZ /100 ); +cond_resched (); }
Reminds me, we have to kill that blk_congestion_wait() stuff too.
Signature · 1 line
Jens Axboe
Am Mo, den 29.03.2004, um 16:12 Uhr -0800, schrieb Andrew Morton:
Quoted 7 lines
How come? Isn't this problem just "gee, we have a lot of stuff to encrypt during writeback"? If so, then it should be sufficient to poke a hole in the encryption loop?
drivers/md/dm-crypt.c static int crypt_map(struct dm_target *t · line 675
Quoted 7 lines
drivers/md/dm-crypt.c renamed from drivers/md/dm-crypt.c~a +1
-0@@ -669,6 +669,7 @@ static int crypt_map(struct dm_target *t/* out of memory -> run queues */ if (remaining)blk_congestion_wait (bio_data_dir (clone),HZ /100 ); +cond_resched (); }/* drop reference, clones could have returned before we reach this */
cryptoapi always does this after every block. It also happens with preemption enabled. I got feedback from a person who said that renicing pdflush to 0 helped. So it looks like the CPU scheduler doesn't want to schedule pdflush away. Hmm.
Quoted 16 lines · Christophe Saout
Am Mo, den 29.03.2004, um 16:12 Uhr -0800, schrieb Andrew Morton:
Quoted 13 lines
How come? Isn't this problem just "gee, we have a lot of stuff to encrypt during writeback"? If so, then it should be sufficient to poke a hole in the encryption loop?
drivers/md/dm-crypt.c renamed from drivers/md/dm-crypt.c~a +1
-0@@ -669,6 +669,7 @@ static int crypt_map(struct dm_target *t/* out of memory -> run queues */ if (remaining)blk_congestion_wait (bio_data_dir (clone),HZ /100 ); +cond_resched (); }
Quoted 6 lines
drivers/md/dm-crypt.c static int crypt_map(struct dm_target *t · line 675
Quoted 1 line
drivers/md/dm-crypt.c renamed from drivers/md/dm-crypt.c~a +0
-0/* drop reference, clones could have returned before we reach this */ cryptoapi always does this after every block. It also happens with preemption enabled. I got feedback from a person who said that renicing pdflush to 0 helped. So it looks like the CPU scheduler doesn't want to schedule pdflush away. Hmm.
Oh, OK, since pdflush was converted to use the kthread stuff it has been running at keventd's `nice -10'. You can probably renice it by hand.
mm/pdflush.c renamed from mm/pdflush.c~pdflush-nice-0 +6 -0
@@ -177,6 +177,12 @@ static int __pdflush(struct pdflush_workstatic int pdflush (void *dummy) {struct pdflush_work my_work; + +/* +* pdflush can spend a lot of time doing encryption via dm-crypt. We +* don't want to do that at keventd's priority. +*/ +set_user_nice (current,0 );return __pdflush (&my_work); }
_
Hello Jens,
Quoted 2 lines
Try the -mm kernels intead, should have lots better behaviour for pdflush/dm interactions.
allright. with -mm5 it is _much_ better. load is back to normal. I suppose we should get all this stuff into mainline ASAP ;) I'm sure lot of people are using this combo.
thanks for you help,
--
Zoltan NAGY,
Network Administrator
Quoted 9 lines · Zoltan NAGY
I've just upgraded the system to 2.6.5-rc2-bk6, and I'm using dm-crypt. It's a heavily used server, on average 20-30mbit/sec traffic is on the wire 7/24, and just noticed, that the load is very high. In every 4-5 sec pdflush takes a lot of cpu... Is this intentional? I've found a similar question on kerneltrap
(http://kerneltrap.org/node/view/2756), but havent found a solution yet. I'm just wondering if it is a problem, or it's the normal behavior? It's a 1.8 P4 with 1G ram and highmem enabled, with 256 bit aes thru dm-crypt.
Same here (duron 1.2GHz, 512MB ram, IDE disks (via and promise controllers)) and it doesn't matter if I'm using cryptoloop or dm-crypt.
System hangs for about one seconds (xmms stops playing, xterms are not responding to key presses) and then everything back to normal until next hang happens (in few seconds).
Quoted 2 lines
Zoltan NAGY,
Network Administrator
ps. I'm not sure (didn't do any checking to be sure) but afaik this started to happening with 2.6.3 kernel.
Signature · 2 lines
Arkadiusz Miśkiewicz CS at FoE, Wroclaw University of Technology arekm.pld-linux.org, 1024/3DB19BBD, JID: arekm.jabber.org, PLD/Linux