Skip to contentSkip to conversation

Reduce GFP_ATOMIC allocation failures, candidate fix V3

PATCH In review Mel Gorman 6 participants 23 messages Nov 12, 2009 – Nov 19, 2009

Subsystems (3)
lore
Share & tools
lore

Mel Gorman

author
#0 lore raw

cover Reduce GFP_ATOMIC allocation failures, candidate fix V3

Sorry for the long delay in posting another version. Testing is extremely time-consuming and I wasn't getting to work on this as much as I'd have liked.

Changelog since V2
o Dropped the kswapd-quickly-notice-high-order patch. In more detailed testing, it made latencies even worse as kswapd slept more on high-order congestion causing order-0 direct reclaims.
o Added changes to how congestion_wait() works
o Added a number of new patches altering the behaviour of reclaim

Since 2.6.31-rc1, there have been an increasing number of GFP_ATOMIC failures. A significant number of these have been high-order GFP_ATOMIC failures and while they are generally brushed away, there has been a large increase in them recently and there are a number of possible areas the problem could be in - core vm, page writeback and a specific driver. The bugs affected by this that I am aware of are;

[Bug #14141] order 2 page allocation failures in iwlagn
[Bug #14141] order 2 page allocation failures (generic)
[Bug #14265] ifconfig: page allocation failure. order:5, mode:0x8020 w/ e100
[No BZ ID] Kernel crash on 2.6.31.x (kcryptd: page allocation failure..)
[No BZ ID] page allocation failure message kernel 2.6.31.4 (tty-related)

The following are a series of patches that bring the behaviour of reclaim and the page allocator more in line with 2.6.30.

Patches 1-3 should be tested first. The testing I've done shows that the page allocator and behaviour of congestion_wait() is more in line with 2.6.30 than the vanilla kernels.

It'd be nice to have 2 more tests, applying each patch on top noting any behaviour change. i.e. ideally there would be results for

o patches 1+2+3
o patches 1+2+3+4
o patches 1+2+3+4+5

Of course, any tests results are welcome. The rest of the mail is the results of my own tests.

I've tested against 2.6.31 and 2.6.32-rc6. I've somewhat replicated the problem in Bug #14141 and believe the other bugs are variations of the same style of problem. The basic reproduction case was;

1. X86-64 AMD Phenom and X86 P4 booted with mem=512MB. Expectation is any machine will do as long as it's 512MB for the size of workload involved.

2. A crypted work partition and swap partition was created. On my own setup, I gave no passphrase so it'd be easier to activate without interaction but there are multiple options. I should have taken better notes but the setup goes something like this;

cryptsetup create -y crypt-partition /dev/sda5 pvcreate /dev/mapper/crypt-partition vgcreate crypt-volume /dev/mapper/crypt-partition lvcreate -L 5G -n crypt-logical crypt-volume lvcreate -L 2G -n crypt-swap crypt-volume mkfs -t ext3 /dev/crypt-volume/crypt-logical mkswap /dev/crypt-volume/crypt-swap

3. With the partition mounted on /scratch, I
cd /scratch
mkdir music
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux-2.6

4. On a normal partition, I expand a tarball containing test scripts available at
http://www.csn.ul.ie/~mel/postings/latency-20091112/latency-tests-with-results.tar.gz

There are two helper programs that run as part of the test - a fake music player and a fake gitk.

The fake music player uses rsync with bandwidth limits to start downloading a music folder from another machine. It's bandwidth limited to simulate playing music over NFS. I believe it generates similar if not exact traffic to a music player. It occured to be afterwards that if one patched ogg123 to print a line when 1/10th of a seconds worth of music was played, it could be used as an indirect measure of desktop interactivity and help pin down pesky "audio skips" bug reports.

The fake gitk is based on observing roughly what gitk does using strace. It loads all the logs into a large buffer and then builds a very basic hash map of parent to child commits. The data is stored because it was insufficient just to read the logs. It had to be kept in an in-memory buffer to generate swap. It then discards the data and does it over again in a loop for a small number of times so the test is finite. When it processes a large number of commits, it outputs a line to stdout so that stalls can be observed. Ideal behaviour is that commits are read at a constant rate and latencies look flat.

Output from the two programs is piped through another script - latency-output. It records how far into the test it was when the line was outputted and what the latency was since the last line appeared. The latency should always be very smooth. Because pipes buffer IO, they are all run by expect_unbuffered which is available from expect-dev on Debian at least.

All the tests are driven via run-test.sh. While the tests run, it records the kern.log to track page allocation failures, records nr_writeback at regular intervals and tracks Page IO and Swap IO.

5. For running an actual test, a kernel is built, booted, the crypted partition activated, lvm restarted, /dev/crypt-volume/crypt-logical mounted on /scratch, all swap partitions turned off and then the swap partition on /dev/crypt-volume/crypt-swap activated. I then run run-test.sh from the tarball

6. Run the test script

To evaluate the patches, I considered three basic metrics.

o The length of time it takes fake-gitk to complete on average
o How often and how long fake-gitk stalled for
o How long was spent in congestion_wait

All generated data is in the tarball.

On X86, the results I got were

2.6.30-0000000-force-highorder Elapsed:10:59.095 Failures:0

2.6.31-0000000-force-highorder           Elapsed:11:53.505  Failures:0
2.6.31-revert-8aa7e847                   Elapsed:14:01.595  Failures:0
2.6.31-0000012-pgalloc-2.6.30            Elapsed:13:32.237  Failures:0
2.6.31-0000123-congestion-both           Elapsed:12:44.170  Failures:0
2.6.31-0001234-kswapd-quick-recheck      Elapsed:10:35.327  Failures:0
2.6.31-0012345-adjust-priority           Elapsed:11:02.995  Failures:0
2.6.32-rc6-0000000-force-highorder       Elapsed:18:18.562  Failures:0
2.6.32-rc6-revert-8aa7e847               Elapsed:10:29.278  Failures:0
2.6.32-rc6-0000012-pgalloc-2.6.30        Elapsed:13:32.393  Failures:0
2.6.32-rc6-0000123-congestion-both       Elapsed:14:55.265  Failures:0
2.6.32-rc6-0001234-kswapd-quick-recheck  Elapsed:13:35.628  Failures:0
2.6.32-rc6-0012345-adjust-priority       Elapsed:12:41.278  Failures:0

The 0000000-force-highorder is a vanilla kernel patched so that network receive always results in an order-2 allocation. This machine wasn't suffering page allocation failures even under this circumstance. However, note how slow 2.6.32-rc6 is and how much the revert helps. With the patches applied, there is comparable performance.

Latencies were generally reduced with the patches applied. 2.6.32-rc6 was particularly crazy with long stalls measured over the duration of the test but has comparable latencies with 2.6.30 with the patches applied.

congestion_wait behaviour is more in line with 2.6.30 after the patches with similar amounts of time being spent. In general, 2.6.32-rc6-0012345-adjust-priority waits for longer than 2.6.30 or the reverted kernels did. It also waits in more instances such as inside shrink_inactive_list() where it didn't before. Forcing behaviour like 2.6.30 resulted in good figures but I couldn't justify the patches with anything more solid than "in tests, it behaves well even though it doesn't make a lot of sense"

On X86-64, the results I got were

2.6.30-0000000-force-highorder Elapsed:09:48.545 Failures:0

2.6.31-0000000-force-highorder           Elapsed:09:13.020  Failures:0
2.6.31-revert-8aa7e847                   Elapsed:09:02.120  Failures:0
2.6.31-0000012-pgalloc-2.6.30            Elapsed:08:52.742  Failures:0
2.6.31-0000123-congestion-both           Elapsed:08:59.375  Failures:0
2.6.31-0001234-kswapd-quick-recheck      Elapsed:09:19.208  Failures:0
2.6.31-0012345-adjust-priority           Elapsed:09:39.225  Failures:0
2.6.32-rc6-0000000-force-highorder       Elapsed:19:38.585  Failures:5
2.6.32-rc6-revert-8aa7e847               Elapsed:17:21.257  Failures:0
2.6.32-rc6-0000012-pgalloc-2.6.30        Elapsed:18:56.682  Failures:1
2.6.32-rc6-0000123-congestion-both       Elapsed:16:08.340  Failures:0
2.6.32-rc6-0001234-kswapd-quick-recheck  Elapsed:18:11.200  Failures:7
2.6.32-rc6-0012345-adjust-priority       Elapsed:21:33.158  Failures:0

Failures were down and my impression was that it was much harder to cause failures. Performance on mainline is still not as good as 2.6.30. On this particular machine, I was able to force performance to be in line but not with any patch I could justify in the general case.

Latencies were slightly reduced by applying the patches against 2.6.31. against 2.6.32-rc6, applying the patches significantly reduced the latencies but they are still significant. I'll continue to investigate what can be done to improve this further.

Again, congestion_wait() is more in line with 2.6.30 when the patches are applied. Similarly to X86, almost identical behaviour can be forced by waiting on BLK_ASYNC_BOTH for each caller to congestion_wait() in the reclaim and allocator paths.

Bottom line, the patches made triggering allocation failures much harder and in a number of instances and latencies are reduced when the system is under load. I will keep looking around this area - particularly the performance under load on 2.6.32-rc6 but with 2.6.32 almost out the door, I am releasing what I have now.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#1 lore raw

PATCH 1/5 page allocator: Always wake kswapd when restarting an allocation attempt after direct reclaim failed

If a direct reclaim makes no forward progress, it considers whether it should go OOM or not. Whether OOM is triggered or not, it may retry the application afterwards. In times past, this would always wake kswapd as well but currently, kswapd is not woken up after direct reclaim fails. For order-0 allocations, this makes little difference but if there is a heavy mix of higher-order allocations that direct reclaim is failing for, it might mean that kswapd is not rewoken for higher orders as much as it did previously.

This patch wakes up kswapd when an allocation is being retried after a direct reclaim failure. It would be expected that kswapd is already awake, but this has the effect of telling kswapd to reclaim at the higher order as well.

---
 mm/page_alloc.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
mm/page_alloc.c +1 -1
index cdcedf6..250d055 100644
@@ -1817,9 +1817,9 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	if (NUMA_BUILD && (gfp_mask & GFP_THISNODE) == GFP_THISNODE)
 		goto nopage;
 
+restart:
 	wake_all_kswapd(order, zonelist, high_zoneidx);
 
-restart:
 	/*
 	 * OK, we're below the kswapd watermark and have kicked background
 	 * reclaim. Now things get more complex, so set up alloc_flags according
Signature · 1 line
1.6.5
#2 lore raw
In reply to Mel Gorman
Quoted 10 lines · Mel Gorman

Sorry for the long delay in posting another version. Testing is extremely time-consuming and I wasn't getting to work on this as much as I'd have liked.

Changelog since V2
o Dropped the kswapd-quickly-notice-high-order patch. In more detailed testing, it made latencies even worse as kswapd slept more on high-order congestion causing order-0 direct reclaims.
o Added changes to how congestion_wait() works
o Added a number of new patches altering the behaviour of reclaim

Quoted 6 lines

Since 2.6.31-rc1, there have been an increasing number of GFP_ATOMIC failures. A significant number of these have been high-order GFP_ATOMIC failures and while they are generally brushed away, there has been a large increase in them recently and there are a number of possible areas the problem could be in - core vm, page writeback and a specific driver. The bugs affected by this that I am aware of are;

Thanks for all the time you've spent on this one. Let me start with some more questions about the workload ;)

Quoted 21 lines

2. A crypted work partition and swap partition was created. On my own setup, I gave no passphrase so it'd be easier to activate without interaction but there are multiple options. I should have taken better notes but the setup goes something like this;

cryptsetup create -y crypt-partition /dev/sda5 pvcreate /dev/mapper/crypt-partition vgcreate crypt-volume /dev/mapper/crypt-partition lvcreate -L 5G -n crypt-logical crypt-volume lvcreate -L 2G -n crypt-swap crypt-volume mkfs -t ext3 /dev/crypt-volume/crypt-logical mkswap /dev/crypt-volume/crypt-swap

3. With the partition mounted on /scratch, I
cd /scratch
mkdir music
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux-2.6

4. On a normal partition, I expand a tarball containing test scripts available at
http://www.csn.ul.ie/~mel/postings/latency-20091112/latency-tests-with-results.tar.gz

Quoted 6 lines

There are two helper programs that run as part of the test - a fake music player and a fake gitk.

The fake music player uses rsync with bandwidth limits to start downloading a music folder from another machine. It's bandwidth limited to simulate playing music over NFS.

So the workload is gitk reading a git repo and a program reading data over the network. Which part of the workload writes to disk?

-chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw
Quoted 18 lines · Chris Mason
Quoted 10 lines · Mel Gorman

Sorry for the long delay in posting another version. Testing is extremely time-consuming and I wasn't getting to work on this as much as I'd have liked.

Changelog since V2
o Dropped the kswapd-quickly-notice-high-order patch. In more detailed testing, it made latencies even worse as kswapd slept more on high-order congestion causing order-0 direct reclaims.
o Added changes to how congestion_wait() works
o Added a number of new patches altering the behaviour of reclaim

Quoted 6 lines

Since 2.6.31-rc1, there have been an increasing number of GFP_ATOMIC failures. A significant number of these have been high-order GFP_ATOMIC failures and while they are generally brushed away, there has been a large increase in them recently and there are a number of possible areas the problem could be in - core vm, page writeback and a specific driver. The bugs affected by this that I am aware of are;

Quoted 5 lines

Thanks for all the time you've spent on this one. Let me start with some more questions about the workload ;)

So the workload is gitk reading a git repo and a program reading data over the network. Which part of the workload writes to disk?

Sorry for the self reply, I started digging through your data (man, that's a lot of data ;). I took another tour through dm-crypt and things make more sense now.

dm-crypt has two different single threaded workqueues for each dm-crypt device. The first one is meant to deal with the actual encryption and decryption, and the second one is meant to do the IO.

So the path for a write looks something like this:

filesystem -> crypt thread -> encrypt the data -> io thread -> disk

And the path for read looks something like this:

filesystem -> io thread -> disk -> crypt thread -> decrypt data -> FS

One thread does encryption and one thread does IO, and these threads are shared for reads and writes. The end result is that all of the sync reads get stuck behind any async write congestion and all of the async writes get stuck behind any sync read congestion.

It's almost like you need to check for both sync and async congestion before you have any hopes of a new IO making progress.

The confusing part is that dm hasn't gotten any worse in this regard since 2.6.30 but the workload here is generating more sync reads (hopefully from gitk and swapin) than async writes (from the low bandwidth rsync). So in general if you were to change mm/*.c wait for sync congestion instead of async, things should appear better.

The punch line is that the btrfs guy thinks we can solve all of this with just one more thread. If we change dm-crypt to have a thread dedicated to sync IO and a thread dedicated to async IO the system should smooth out.

-chris

#4 lore raw
In reply to Chris Mason
Quoted 18 lines · Chris Mason
Quoted 10 lines · Mel Gorman

Sorry for the long delay in posting another version. Testing is extremely time-consuming and I wasn't getting to work on this as much as I'd have liked.

Changelog since V2
o Dropped the kswapd-quickly-notice-high-order patch. In more detailed testing, it made latencies even worse as kswapd slept more on high-order congestion causing order-0 direct reclaims.
o Added changes to how congestion_wait() works
o Added a number of new patches altering the behaviour of reclaim

Quoted 6 lines

Since 2.6.31-rc1, there have been an increasing number of GFP_ATOMIC failures. A significant number of these have been high-order GFP_ATOMIC failures and while they are generally brushed away, there has been a large increase in them recently and there are a number of possible areas the problem could be in - core vm, page writeback and a specific driver. The bugs affected by this that I am aware of are;

Quoted 5 lines

Thanks for all the time you've spent on this one. Let me start with some more questions about the workload ;)

So the workload is gitk reading a git repo and a program reading data over the network. Which part of the workload writes to disk?

Sorry for the self reply, I started digging through your data (man, that's a lot of data ;). I took another tour through dm-crypt and things make more sense now.

dm-crypt has two different single threaded workqueues for each dm-crypt device. The first one is meant to deal with the actual encryption and decryption, and the second one is meant to do the IO.

So the path for a write looks something like this:

filesystem -> crypt thread -> encrypt the data -> io thread -> disk

And the path for read looks something like this:

filesystem -> io thread -> disk -> crypt thread -> decrypt data -> FS

One thread does encryption and one thread does IO, and these threads are shared for reads and writes. The end result is that all of the sync reads get stuck behind any async write congestion and all of the async writes get stuck behind any sync read congestion.

It's almost like you need to check for both sync and async congestion before you have any hopes of a new IO making progress.

The confusing part is that dm hasn't gotten any worse in this regard since 2.6.30 but the workload here is generating more sync reads (hopefully from gitk and swapin) than async writes (from the low bandwidth rsync). So in general if you were to change mm/*.c wait for sync congestion instead of async, things should appear better.

The punch line is that the btrfs guy thinks we can solve all of this with just one more thread. If we change dm-crypt to have a thread dedicated to sync IO and a thread dedicated to async IO the system should smooth out.

-chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#5 lore raw

On Thu, Nov 12, 2009 at 05:00:05PM -0500, Chris Mason wrote:

[ ...]

Quoted 5 lines

The punch line is that the btrfs guy thinks we can solve all of this with just one more thread. If we change dm-crypt to have a thread dedicated to sync IO and a thread dedicated to async IO the system should smooth out.

This is pretty likely to set your dm data on fire. It's only for Mel who starts his script w/mkfs.

It adds the second thread and more importantly makes sure the kcryptd thread doesn't get stuck waiting for requests.

-chris

drivers/md/dm-crypt.c +15 -3
index ed10381..295ffeb 100644
@@ -94,6 +94,7 @@ struct crypt_config {
 	struct bio_set *bs;
 
 	struct workqueue_struct *io_queue;
+	struct workqueue_struct *async_io_queue;
 	struct workqueue_struct *crypt_queue;
 
 	/*
@@ -691,7 +692,10 @@ static void kcryptd_queue_io(struct dm_crypt_io *io)
 	struct crypt_config *cc = io->target->private;
 
 	INIT_WORK(&io->work, kcryptd_io);
-	queue_work(cc->io_queue, &io->work);
+	if (io->base_bio->bi_rw & (1 << BIO_RW_SYNCIO))
+		queue_work(cc->io_queue, &io->work);
+	else
+		queue_work(cc->async_io_queue, &io->work);
 }
 
 static void kcryptd_crypt_write_io_submit(struct dm_crypt_io *io,
@@ -759,8 +763,7 @@ static void kcryptd_crypt_write_convert(struct dm_crypt_io *io)
 
 		/* Encryption was already finished, submit io now */
 		if (crypt_finished) {
-			kcryptd_crypt_write_io_submit(io, r, 0);
-
+			kcryptd_crypt_write_io_submit(io, r, 1);
 			/*
 			 * If there was an error, do not try next fragments.
 			 * For async, error is processed in async handler.
@@ -1120,6 +1123,12 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
 	} else
 		cc->iv_mode = NULL;
 
+	cc->async_io_queue = create_singlethread_workqueue("kcryptd_async_io");
+	if (!cc->async_io_queue) {
+		ti->error = "Couldn't create kcryptd io queue";
+		goto bad_async_io_queue;
+	}
+
 	cc->io_queue = create_singlethread_workqueue("kcryptd_io");
 	if (!cc->io_queue) {
 		ti->error = "Couldn't create kcryptd io queue";
@@ -1139,6 +1148,8 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
 bad_crypt_queue:
 	destroy_workqueue(cc->io_queue);
 bad_io_queue:
+	destroy_workqueue(cc->async_io_queue);
+bad_async_io_queue:
 	kfree(cc->iv_mode);
 bad_ivmode_string:
 	dm_put_device(ti, cc->dev);
@@ -1166,6 +1177,7 @@ static void crypt_dtr(struct dm_target *ti)
 	struct crypt_config *cc = (struct crypt_config *) ti->private;
 
 	destroy_workqueue(cc->io_queue);
+	destroy_workqueue(cc->async_io_queue);
 	destroy_workqueue(cc->crypt_queue);
 
 	if (cc->req)
#6 lore raw
In reply to Chris Mason

On Thu, Nov 12, 2009 at 05:00:05PM -0500, Chris Mason wrote:

[ ...]

Quoted 5 lines

The punch line is that the btrfs guy thinks we can solve all of this with just one more thread. If we change dm-crypt to have a thread dedicated to sync IO and a thread dedicated to async IO the system should smooth out.

This is pretty likely to set your dm data on fire. It's only for Mel who starts his script w/mkfs.

It adds the second thread and more importantly makes sure the kcryptd thread doesn't get stuck waiting for requests.

-chris

drivers/md/dm-crypt.c +15 -3
index ed10381..295ffeb 100644
@@ -94,6 +94,7 @@ struct crypt_config {
 	struct bio_set *bs;
 
 	struct workqueue_struct *io_queue;
+	struct workqueue_struct *async_io_queue;
 	struct workqueue_struct *crypt_queue;
 
 	/*
@@ -691,7 +692,10 @@ static void kcryptd_queue_io(struct dm_crypt_io *io)
 	struct crypt_config *cc = io->target->private;
 
 	INIT_WORK(&io->work, kcryptd_io);
-	queue_work(cc->io_queue, &io->work);
+	if (io->base_bio->bi_rw & (1 << BIO_RW_SYNCIO))
+		queue_work(cc->io_queue, &io->work);
+	else
+		queue_work(cc->async_io_queue, &io->work);
 }
 
 static void kcryptd_crypt_write_io_submit(struct dm_crypt_io *io,
@@ -759,8 +763,7 @@ static void kcryptd_crypt_write_convert(struct dm_crypt_io *io)
 
 		/* Encryption was already finished, submit io now */
 		if (crypt_finished) {
-			kcryptd_crypt_write_io_submit(io, r, 0);
-
+			kcryptd_crypt_write_io_submit(io, r, 1);
 			/*
 			 * If there was an error, do not try next fragments.
 			 * For async, error is processed in async handler.
@@ -1120,6 +1123,12 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
 	} else
 		cc->iv_mode = NULL;
 
+	cc->async_io_queue = create_singlethread_workqueue("kcryptd_async_io");
+	if (!cc->async_io_queue) {
+		ti->error = "Couldn't create kcryptd io queue";
+		goto bad_async_io_queue;
+	}
+
 	cc->io_queue = create_singlethread_workqueue("kcryptd_io");
 	if (!cc->io_queue) {
 		ti->error = "Couldn't create kcryptd io queue";
@@ -1139,6 +1148,8 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
 bad_crypt_queue:
 	destroy_workqueue(cc->io_queue);
 bad_io_queue:
+	destroy_workqueue(cc->async_io_queue);
+bad_async_io_queue:
 	kfree(cc->iv_mode);
 bad_ivmode_string:
 	dm_put_device(ti, cc->dev);
@@ -1166,6 +1177,7 @@ static void crypt_dtr(struct dm_target *ti)
 	struct crypt_config *cc = (struct crypt_config *) ti->private;
 
 	destroy_workqueue(cc->io_queue);
+	destroy_workqueue(cc->async_io_queue);
 	destroy_workqueue(cc->crypt_queue);
 
 	if (cc->req)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#7 lore raw

PATCH 1/1 make crypto unplug fix V3

This is still likely to set your dm data on fire. It is only meant for testers that start with mkfs and don't have any valuable dm data.

It includes my patch from last night, along with changes to force dm to unplug when its IO queues empty.

The problem goes like this:

Process: submit read bio
dm: put bio onto work queue
process: unplug
dm: work queue finds bio, does a generic_make_request

The end result is that we miss the unplug completely. dm-crypt needs to unplug for sync bios. This patch also changes it to unplug whenever the queue is empty, which is far from ideal but better than missing the unplugs.

This doesn't completely fix io stalls I'm seeing with dm-crypt, but its my best guess. If it works, I'll break it up and submit for real to the dm people.

-chris

drivers/md/dm-crypt.c +42 -3
index ed10381..729ae01 100644
@@ -94,8 +94,12 @@ struct crypt_config {
 	struct bio_set *bs;
 
 	struct workqueue_struct *io_queue;
+	struct workqueue_struct *async_io_queue;
 	struct workqueue_struct *crypt_queue;
 
+	atomic_t sync_bios_in_queue;
+	atomic_t async_bios_in_queue;
+
 	/*
 	 * crypto related data
 	 */
@@ -679,11 +683,29 @@ static void kcryptd_io_write(struct dm_crypt_io *io)
 static void kcryptd_io(struct work_struct *work)
 {
 	struct dm_crypt_io *io = container_of(work, struct dm_crypt_io, work);
+	struct crypt_config *cc = io->target->private;
+	int zero_sync = 0;
+	int zero_async = 0;
+	int was_sync = 0;
+
+	if (io->base_bio->bi_rw & (1 << BIO_RW_SYNCIO)) {
+		zero_sync = atomic_dec_and_test(&cc->sync_bios_in_queue);
+		was_sync = 1;
+	} else
+		zero_async = atomic_dec_and_test(&cc->async_bios_in_queue);
 
 	if (bio_data_dir(io->base_bio) == READ)
 		kcryptd_io_read(io);
 	else
 		kcryptd_io_write(io);
+
+	if ((was_sync && zero_sync) ||
+	    (!was_sync && zero_async &&
+	     atomic_read(&cc->sync_bios_in_queue) == 0)) {
+		struct backing_dev_info *bdi;
+		bdi = blk_get_backing_dev_info(io->base_bio->bi_bdev);
+		blk_run_backing_dev(bdi, NULL);
+	}
 }
 
 static void kcryptd_queue_io(struct dm_crypt_io *io)
@@ -691,7 +713,13 @@ static void kcryptd_queue_io(struct dm_crypt_io *io)
 	struct crypt_config *cc = io->target->private;
 
 	INIT_WORK(&io->work, kcryptd_io);
-	queue_work(cc->io_queue, &io->work);
+	if (io->base_bio->bi_rw & (1 << BIO_RW_SYNCIO)) {
+		atomic_inc(&cc->sync_bios_in_queue);
+		queue_work(cc->io_queue, &io->work);
+	} else {
+		atomic_inc(&cc->async_bios_in_queue);
+		queue_work(cc->async_io_queue, &io->work);
+	}
 }
 
 static void kcryptd_crypt_write_io_submit(struct dm_crypt_io *io,
@@ -759,8 +787,7 @@ static void kcryptd_crypt_write_convert(struct dm_crypt_io *io)
 
 		/* Encryption was already finished, submit io now */
 		if (crypt_finished) {
-			kcryptd_crypt_write_io_submit(io, r, 0);
-
+			kcryptd_crypt_write_io_submit(io, r, 1);
 			/*
 			 * If there was an error, do not try next fragments.
 			 * For async, error is processed in async handler.
@@ -1120,6 +1147,15 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
 	} else
 		cc->iv_mode = NULL;
 
+	atomic_set(&cc->sync_bios_in_queue, 0);
+	atomic_set(&cc->async_bios_in_queue, 0);
+
+	cc->async_io_queue = create_singlethread_workqueue("kcryptd_async_io");
+	if (!cc->async_io_queue) {
+		ti->error = "Couldn't create kcryptd io queue";
+		goto bad_async_io_queue;
+	}
+
 	cc->io_queue = create_singlethread_workqueue("kcryptd_io");
 	if (!cc->io_queue) {
 		ti->error = "Couldn't create kcryptd io queue";
@@ -1139,6 +1175,8 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
 bad_crypt_queue:
 	destroy_workqueue(cc->io_queue);
 bad_io_queue:
+	destroy_workqueue(cc->async_io_queue);
+bad_async_io_queue:
 	kfree(cc->iv_mode);
 bad_ivmode_string:
 	dm_put_device(ti, cc->dev);
@@ -1166,6 +1204,7 @@ static void crypt_dtr(struct dm_target *ti)
 	struct crypt_config *cc = (struct crypt_config *) ti->private;
 
 	destroy_workqueue(cc->io_queue);
+	destroy_workqueue(cc->async_io_queue);
 	destroy_workqueue(cc->crypt_queue);
 
 	if (cc->req)
#8 lore raw

PATCH 1/1 make crypto unplug fix V3

In reply to Chris Mason

This is still likely to set your dm data on fire. It is only meant for testers that start with mkfs and don't have any valuable dm data.

It includes my patch from last night, along with changes to force dm to unplug when its IO queues empty.

The problem goes like this:

Process: submit read bio
dm: put bio onto work queue
process: unplug
dm: work queue finds bio, does a generic_make_request

The end result is that we miss the unplug completely. dm-crypt needs to unplug for sync bios. This patch also changes it to unplug whenever the queue is empty, which is far from ideal but better than missing the unplugs.

This doesn't completely fix io stalls I'm seeing with dm-crypt, but its my best guess. If it works, I'll break it up and submit for real to the dm people.

-chris

drivers/md/dm-crypt.c +42 -3
index ed10381..729ae01 100644
@@ -94,8 +94,12 @@ struct crypt_config {
 	struct bio_set *bs;
 
 	struct workqueue_struct *io_queue;
+	struct workqueue_struct *async_io_queue;
 	struct workqueue_struct *crypt_queue;
 
+	atomic_t sync_bios_in_queue;
+	atomic_t async_bios_in_queue;
+
 	/*
 	 * crypto related data
 	 */
@@ -679,11 +683,29 @@ static void kcryptd_io_write(struct dm_crypt_io *io)
 static void kcryptd_io(struct work_struct *work)
 {
 	struct dm_crypt_io *io = container_of(work, struct dm_crypt_io, work);
+	struct crypt_config *cc = io->target->private;
+	int zero_sync = 0;
+	int zero_async = 0;
+	int was_sync = 0;
+
+	if (io->base_bio->bi_rw & (1 << BIO_RW_SYNCIO)) {
+		zero_sync = atomic_dec_and_test(&cc->sync_bios_in_queue);
+		was_sync = 1;
+	} else
+		zero_async = atomic_dec_and_test(&cc->async_bios_in_queue);
 
 	if (bio_data_dir(io->base_bio) == READ)
 		kcryptd_io_read(io);
 	else
 		kcryptd_io_write(io);
+
+	if ((was_sync && zero_sync) ||
+	    (!was_sync && zero_async &&
+	     atomic_read(&cc->sync_bios_in_queue) == 0)) {
+		struct backing_dev_info *bdi;
+		bdi = blk_get_backing_dev_info(io->base_bio->bi_bdev);
+		blk_run_backing_dev(bdi, NULL);
+	}
 }
 
 static void kcryptd_queue_io(struct dm_crypt_io *io)
@@ -691,7 +713,13 @@ static void kcryptd_queue_io(struct dm_crypt_io *io)
 	struct crypt_config *cc = io->target->private;
 
 	INIT_WORK(&io->work, kcryptd_io);
-	queue_work(cc->io_queue, &io->work);
+	if (io->base_bio->bi_rw & (1 << BIO_RW_SYNCIO)) {
+		atomic_inc(&cc->sync_bios_in_queue);
+		queue_work(cc->io_queue, &io->work);
+	} else {
+		atomic_inc(&cc->async_bios_in_queue);
+		queue_work(cc->async_io_queue, &io->work);
+	}
 }
 
 static void kcryptd_crypt_write_io_submit(struct dm_crypt_io *io,
@@ -759,8 +787,7 @@ static void kcryptd_crypt_write_convert(struct dm_crypt_io *io)
 
 		/* Encryption was already finished, submit io now */
 		if (crypt_finished) {
-			kcryptd_crypt_write_io_submit(io, r, 0);
-
+			kcryptd_crypt_write_io_submit(io, r, 1);
 			/*
 			 * If there was an error, do not try next fragments.
 			 * For async, error is processed in async handler.
@@ -1120,6 +1147,15 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
 	} else
 		cc->iv_mode = NULL;
 
+	atomic_set(&cc->sync_bios_in_queue, 0);
+	atomic_set(&cc->async_bios_in_queue, 0);
+
+	cc->async_io_queue = create_singlethread_workqueue("kcryptd_async_io");
+	if (!cc->async_io_queue) {
+		ti->error = "Couldn't create kcryptd io queue";
+		goto bad_async_io_queue;
+	}
+
 	cc->io_queue = create_singlethread_workqueue("kcryptd_io");
 	if (!cc->io_queue) {
 		ti->error = "Couldn't create kcryptd io queue";
@@ -1139,6 +1175,8 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
 bad_crypt_queue:
 	destroy_workqueue(cc->io_queue);
 bad_io_queue:
+	destroy_workqueue(cc->async_io_queue);
+bad_async_io_queue:
 	kfree(cc->iv_mode);
 bad_ivmode_string:
 	dm_put_device(ti, cc->dev);
@@ -1166,6 +1204,7 @@ static void crypt_dtr(struct dm_target *ti)
 	struct crypt_config *cc = (struct crypt_config *) ti->private;
 
 	destroy_workqueue(cc->io_queue);
+	destroy_workqueue(cc->async_io_queue);
 	destroy_workqueue(cc->crypt_queue);
 
 	if (cc->req)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#9 lore raw

Re: [PATCH] make crypto unplug fix V3

Quoted 3 lines · Chris Mason

This is still likely to set your dm data on fire. It is only meant for testers that start with mkfs and don't have any valuable dm data.

The good news is that my room remains fire-free. Despite swap also running from dm-crypt, I had no corruption or instability issues.

Here is an updated set of results for fake-gitk running.

X86

2.6.30-0000000-force-highorder           Elapsed:12:08.908    Failures:0
2.6.31-0000000-force-highorder           Elapsed:10:56.283    Failures:0
2.6.31-0000006-dm-crypt-unplug           Elapsed:11:51.653    Failures:0
2.6.31-0000012-pgalloc-2.6.30            Elapsed:12:26.587    Failures:0
2.6.31-0000123-congestion-both           Elapsed:10:55.298    Failures:0
2.6.31-0001234-kswapd-quick-recheck      Elapsed:18:01.523    Failures:0
2.6.31-0123456-dm-crypt-unplug           Elapsed:10:45.720    Failures:0
2.6.31-revert-8aa7e847                   Elapsed:15:08.020    Failures:0
2.6.32-rc6-0000000-force-highorder       Elapsed:16:20.765    Failures:4
2.6.32-rc6-0000006-dm-crypt-unplug       Elapsed:13:42.920    Failures:0
2.6.32-rc6-0000012-pgalloc-2.6.30        Elapsed:16:13.380    Failures:1
2.6.32-rc6-0000123-congestion-both       Elapsed:18:39.118    Failures:0
2.6.32-rc6-0001234-kswapd-quick-recheck  Elapsed:15:04.398    Failures:0
2.6.32-rc6-0123456-dm-crypt-unplug       Elapsed:12:50.438    Failures:0
2.6.32-rc6-revert-8aa7e847               Elapsed:20:50.888    Failures:0

X86-64

2.6.30-0000000-force-highorder           Elapsed:10:37.300    Failures:0
2.6.31-0000000-force-highorder           Elapsed:08:49.338    Failures:0
2.6.31-0000006-dm-crypt-unplug           Elapsed:09:37.840    Failures:0
2.6.31-0000012-pgalloc-2.6.30            Elapsed:15:49.690    Failures:0
2.6.31-0000123-congestion-both           Elapsed:09:18.790    Failures:0
2.6.31-0001234-kswapd-quick-recheck      Elapsed:08:39.268    Failures:0
2.6.31-0123456-dm-crypt-unplug           Elapsed:08:20.965    Failures:0
2.6.31-revert-8aa7e847                   Elapsed:08:07.457    Failures:0
2.6.32-rc6-0000000-force-highorder       Elapsed:18:29.103    Failures:1
2.6.32-rc6-0000006-dm-crypt-unplug       Elapsed:25:53.515    Failures:3
2.6.32-rc6-0000012-pgalloc-2.6.30        Elapsed:19:55.570    Failures:6
2.6.32-rc6-0000123-congestion-both       Elapsed:17:29.255    Failures:2
2.6.32-rc6-0001234-kswapd-quick-recheck  Elapsed:14:41.068    Failures:0
2.6.32-rc6-0123456-dm-crypt-unplug       Elapsed:15:48.028    Failures:1
2.6.32-rc6-revert-8aa7e847               Elapsed:14:48.647    Failures:0

The numbering in the kernel indicates what patches are applied. I tested the dm-crypt patch both in isolation and in combination with the patches in this series.

Basically, the dm-crypt-unplug makes a small difference in performance overall, mostly slight gains and losses. There was one massive regression with the dm-crypt patch applied to 2.6.32-rc6 but at the moment, I don't know what that is.

In general, the patch reduces the amount of time direct reclaimers are spending on congestion_wait.

Quoted 10 lines

It includes my patch from last night, along with changes to force dm to unplug when its IO queues empty.

The problem goes like this:

Process: submit read bio
dm: put bio onto work queue
process: unplug
dm: work queue finds bio, does a generic_make_request

Quoted 9 lines

The end result is that we miss the unplug completely. dm-crypt needs to unplug for sync bios. This patch also changes it to unplug whenever the queue is empty, which is far from ideal but better than missing the unplugs.

This doesn't completely fix io stalls I'm seeing with dm-crypt, but its my best guess. If it works, I'll break it up and submit for real to the dm people.

Out of curiousity, how are you measuring IO stalls? In the tests I'm doing, the worker processes output their progress and it should be at a steady rate. I considered a stall to be an excessive delay between updates which is a pretty indirect measure.

Quoted 108 lines
(quoted diff) +0 -1
-chris
drivers/md/dm-crypt.c +42 -3
index ed10381..729ae01 100644
@@ -94,8 +94,12 @@ struct crypt_config {
 	struct bio_set *bs;
 
 	struct workqueue_struct *io_queue;
+	struct workqueue_struct *async_io_queue;
 	struct workqueue_struct *crypt_queue;
 
+	atomic_t sync_bios_in_queue;
+	atomic_t async_bios_in_queue;
+
 	/*
 	 * crypto related data
 	 */
@@ -679,11 +683,29 @@ static void kcryptd_io_write(struct dm_crypt_io *io)
 static void kcryptd_io(struct work_struct *work)
 {
 	struct dm_crypt_io *io = container_of(work, struct dm_crypt_io, work);
+	struct crypt_config *cc = io->target->private;
+	int zero_sync = 0;
+	int zero_async = 0;
+	int was_sync = 0;
+
+	if (io->base_bio->bi_rw & (1 << BIO_RW_SYNCIO)) {
+		zero_sync = atomic_dec_and_test(&cc->sync_bios_in_queue);
+		was_sync = 1;
+	} else
+		zero_async = atomic_dec_and_test(&cc->async_bios_in_queue);
 
 	if (bio_data_dir(io->base_bio) == READ)
 		kcryptd_io_read(io);
 	else
 		kcryptd_io_write(io);
+
+	if ((was_sync && zero_sync) ||
+	    (!was_sync && zero_async &&
+	     atomic_read(&cc->sync_bios_in_queue) == 0)) {
+		struct backing_dev_info *bdi;
+		bdi = blk_get_backing_dev_info(io->base_bio->bi_bdev);
+		blk_run_backing_dev(bdi, NULL);
+	}
 }
 
 static void kcryptd_queue_io(struct dm_crypt_io *io)
@@ -691,7 +713,13 @@ static void kcryptd_queue_io(struct dm_crypt_io *io)
 	struct crypt_config *cc = io->target->private;
 
 	INIT_WORK(&io->work, kcryptd_io);
-	queue_work(cc->io_queue, &io->work);
+	if (io->base_bio->bi_rw & (1 << BIO_RW_SYNCIO)) {
+		atomic_inc(&cc->sync_bios_in_queue);
+		queue_work(cc->io_queue, &io->work);
+	} else {
+		atomic_inc(&cc->async_bios_in_queue);
+		queue_work(cc->async_io_queue, &io->work);
+	}
 }
 
 static void kcryptd_crypt_write_io_submit(struct dm_crypt_io *io,
@@ -759,8 +787,7 @@ static void kcryptd_crypt_write_convert(struct dm_crypt_io *io)
 
 		/* Encryption was already finished, submit io now */
 		if (crypt_finished) {
-			kcryptd_crypt_write_io_submit(io, r, 0);
-
+			kcryptd_crypt_write_io_submit(io, r, 1);
 			/*
 			 * If there was an error, do not try next fragments.
 			 * For async, error is processed in async handler.
@@ -1120,6 +1147,15 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
 	} else
 		cc->iv_mode = NULL;
 
+	atomic_set(&cc->sync_bios_in_queue, 0);
+	atomic_set(&cc->async_bios_in_queue, 0);
+
+	cc->async_io_queue = create_singlethread_workqueue("kcryptd_async_io");
+	if (!cc->async_io_queue) {
+		ti->error = "Couldn't create kcryptd io queue";
+		goto bad_async_io_queue;
+	}
+
 	cc->io_queue = create_singlethread_workqueue("kcryptd_io");
 	if (!cc->io_queue) {
 		ti->error = "Couldn't create kcryptd io queue";
@@ -1139,6 +1175,8 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
 bad_crypt_queue:
 	destroy_workqueue(cc->io_queue);
 bad_io_queue:
+	destroy_workqueue(cc->async_io_queue);
+bad_async_io_queue:
 	kfree(cc->iv_mode);
 bad_ivmode_string:
 	dm_put_device(ti, cc->dev);
@@ -1166,6 +1204,7 @@ static void crypt_dtr(struct dm_target *ti)
 	struct crypt_config *cc = (struct crypt_config *) ti->private;
 
 	destroy_workqueue(cc->io_queue);
+	destroy_workqueue(cc->async_io_queue);
 	destroy_workqueue(cc->crypt_queue);
 
 	if (cc->req)
Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#10 lore raw

Re: [PATCH] make crypto unplug fix V3

Quoted 7 lines · Mel Gorman
Quoted 3 lines · Chris Mason

This is still likely to set your dm data on fire. It is only meant for testers that start with mkfs and don't have any valuable dm data.

The good news is that my room remains fire-free. Despite swap also running from dm-crypt, I had no corruption or instability issues.

Ok, definitely not so convincing I'd try and shove it into a late rc.

Quoted 37 lines

Here is an updated set of results for fake-gitk running.

X86

2.6.30-0000000-force-highorder           Elapsed:12:08.908    Failures:0
2.6.31-0000000-force-highorder           Elapsed:10:56.283    Failures:0
2.6.31-0000006-dm-crypt-unplug           Elapsed:11:51.653    Failures:0
2.6.31-0000012-pgalloc-2.6.30            Elapsed:12:26.587    Failures:0
2.6.31-0000123-congestion-both           Elapsed:10:55.298    Failures:0
2.6.31-0001234-kswapd-quick-recheck      Elapsed:18:01.523    Failures:0
2.6.31-0123456-dm-crypt-unplug           Elapsed:10:45.720    Failures:0
2.6.31-revert-8aa7e847                   Elapsed:15:08.020    Failures:0
2.6.32-rc6-0000000-force-highorder       Elapsed:16:20.765    Failures:4
2.6.32-rc6-0000006-dm-crypt-unplug       Elapsed:13:42.920    Failures:0
2.6.32-rc6-0000012-pgalloc-2.6.30        Elapsed:16:13.380    Failures:1
2.6.32-rc6-0000123-congestion-both       Elapsed:18:39.118    Failures:0
2.6.32-rc6-0001234-kswapd-quick-recheck  Elapsed:15:04.398    Failures:0
2.6.32-rc6-0123456-dm-crypt-unplug       Elapsed:12:50.438    Failures:0
2.6.32-rc6-revert-8aa7e847               Elapsed:20:50.888    Failures:0

X86-64

2.6.30-0000000-force-highorder           Elapsed:10:37.300    Failures:0
2.6.31-0000000-force-highorder           Elapsed:08:49.338    Failures:0
2.6.31-0000006-dm-crypt-unplug           Elapsed:09:37.840    Failures:0
2.6.31-0000012-pgalloc-2.6.30            Elapsed:15:49.690    Failures:0
2.6.31-0000123-congestion-both           Elapsed:09:18.790    Failures:0
2.6.31-0001234-kswapd-quick-recheck      Elapsed:08:39.268    Failures:0
2.6.31-0123456-dm-crypt-unplug           Elapsed:08:20.965    Failures:0
2.6.31-revert-8aa7e847                   Elapsed:08:07.457    Failures:0
2.6.32-rc6-0000000-force-highorder       Elapsed:18:29.103    Failures:1
2.6.32-rc6-0000006-dm-crypt-unplug       Elapsed:25:53.515    Failures:3
2.6.32-rc6-0000012-pgalloc-2.6.30        Elapsed:19:55.570    Failures:6
2.6.32-rc6-0000123-congestion-both       Elapsed:17:29.255    Failures:2
2.6.32-rc6-0001234-kswapd-quick-recheck  Elapsed:14:41.068    Failures:0
2.6.32-rc6-0123456-dm-crypt-unplug       Elapsed:15:48.028    Failures:1
2.6.32-rc6-revert-8aa7e847               Elapsed:14:48.647    Failures:0
Quoted 8 lines

The numbering in the kernel indicates what patches are applied. I tested the dm-crypt patch both in isolation and in combination with the patches in this series.

Basically, the dm-crypt-unplug makes a small difference in performance overall, mostly slight gains and losses. There was one massive regression with the dm-crypt patch applied to 2.6.32-rc6 but at the moment, I don't know what that is.

How consistent are your numbers between runs? I was trying to match this up with your last email and things were pretty different.

Quoted 24 lines

In general, the patch reduces the amount of time direct reclaimers are spending on congestion_wait.

Quoted 10 lines

It includes my patch from last night, along with changes to force dm to unplug when its IO queues empty.

The problem goes like this:

Process: submit read bio
dm: put bio onto work queue
process: unplug
dm: work queue finds bio, does a generic_make_request

Quoted 9 lines

The end result is that we miss the unplug completely. dm-crypt needs to unplug for sync bios. This patch also changes it to unplug whenever the queue is empty, which is far from ideal but better than missing the unplugs.

This doesn't completely fix io stalls I'm seeing with dm-crypt, but its my best guess. If it works, I'll break it up and submit for real to the dm people.

Quoted 4 lines

Out of curiousity, how are you measuring IO stalls? In the tests I'm doing, the worker processes output their progress and it should be at a steady rate. I considered a stall to be an excessive delay between updates which is a pretty indirect measure.

I just setup a crypto disk and did dd if=/dev/zero of=/mnt/foo bs=1M

If you watch vmstat 1, there's supposed to be a constant steam of IO to the disk. If a whole second goes by with zero IO, we're doing something wrong, I get a number of multi-second stalls where we are just waiting for IO to happen.

Most of the time I was able to catch a sysrq-w for it, someone was
waiting on a read to finish. It isn't completely clear to me if the
unplugging is working properly.

-chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#11 lore raw

Re: [PATCH] make crypto unplug fix V3

Quoted 52 lines · Chris Mason
Quoted 7 lines · Mel Gorman
Quoted 3 lines · Chris Mason

This is still likely to set your dm data on fire. It is only meant for testers that start with mkfs and don't have any valuable dm data.

The good news is that my room remains fire-free. Despite swap also running from dm-crypt, I had no corruption or instability issues.

Ok, definitely not so convincing I'd try and shove it into a late rc.

Quoted 41 lines

Here is an updated set of results for fake-gitk running.

X86

2.6.30-0000000-force-highorder           Elapsed:12:08.908    Failures:0
2.6.31-0000000-force-highorder           Elapsed:10:56.283    Failures:0
2.6.31-0000006-dm-crypt-unplug           Elapsed:11:51.653    Failures:0
2.6.31-0000012-pgalloc-2.6.30            Elapsed:12:26.587    Failures:0
2.6.31-0000123-congestion-both           Elapsed:10:55.298    Failures:0
2.6.31-0001234-kswapd-quick-recheck      Elapsed:18:01.523    Failures:0
2.6.31-0123456-dm-crypt-unplug           Elapsed:10:45.720    Failures:0
2.6.31-revert-8aa7e847                   Elapsed:15:08.020    Failures:0
2.6.32-rc6-0000000-force-highorder       Elapsed:16:20.765    Failures:4
2.6.32-rc6-0000006-dm-crypt-unplug       Elapsed:13:42.920    Failures:0
2.6.32-rc6-0000012-pgalloc-2.6.30        Elapsed:16:13.380    Failures:1
2.6.32-rc6-0000123-congestion-both       Elapsed:18:39.118    Failures:0
2.6.32-rc6-0001234-kswapd-quick-recheck  Elapsed:15:04.398    Failures:0
2.6.32-rc6-0123456-dm-crypt-unplug       Elapsed:12:50.438    Failures:0
2.6.32-rc6-revert-8aa7e847               Elapsed:20:50.888    Failures:0

X86-64

2.6.30-0000000-force-highorder           Elapsed:10:37.300    Failures:0
2.6.31-0000000-force-highorder           Elapsed:08:49.338    Failures:0
2.6.31-0000006-dm-crypt-unplug           Elapsed:09:37.840    Failures:0
2.6.31-0000012-pgalloc-2.6.30            Elapsed:15:49.690    Failures:0
2.6.31-0000123-congestion-both           Elapsed:09:18.790    Failures:0
2.6.31-0001234-kswapd-quick-recheck      Elapsed:08:39.268    Failures:0
2.6.31-0123456-dm-crypt-unplug           Elapsed:08:20.965    Failures:0
2.6.31-revert-8aa7e847                   Elapsed:08:07.457    Failures:0
2.6.32-rc6-0000000-force-highorder       Elapsed:18:29.103    Failures:1
2.6.32-rc6-0000006-dm-crypt-unplug       Elapsed:25:53.515    Failures:3
2.6.32-rc6-0000012-pgalloc-2.6.30        Elapsed:19:55.570    Failures:6
2.6.32-rc6-0000123-congestion-both       Elapsed:17:29.255    Failures:2
2.6.32-rc6-0001234-kswapd-quick-recheck  Elapsed:14:41.068    Failures:0
2.6.32-rc6-0123456-dm-crypt-unplug       Elapsed:15:48.028    Failures:1
2.6.32-rc6-revert-8aa7e847               Elapsed:14:48.647    Failures:0

The numbering in the kernel indicates what patches are applied. I tested the dm-crypt patch both in isolation and in combination with the patches in this series.

Quoted 8 lines
Quoted 4 lines

Basically, the dm-crypt-unplug makes a small difference in performance overall, mostly slight gains and losses. There was one massive regression with the dm-crypt patch applied to 2.6.32-rc6 but at the moment, I don't know what that is.

How consistent are your numbers between runs? I was trying to match this up with your last email and things were pretty different.

The figures from the first mail were based on kernels that were not instrumented. It so happened that this run was based on an instrumented kernel to get the congestion_wait figures so the results are different.

However, the results vary a lot. In some cases, it will spike just as 2.6.32-rc6-0000006-dm-crypt-unplug did. The reported figure is the average of 4 fake-gitk runs. I don't have the standard deviation handy but it's high.

Quoted 29 lines
Quoted 24 lines

In general, the patch reduces the amount of time direct reclaimers are spending on congestion_wait.

Quoted 10 lines

It includes my patch from last night, along with changes to force dm to unplug when its IO queues empty.

The problem goes like this:

Process: submit read bio
dm: put bio onto work queue
process: unplug
dm: work queue finds bio, does a generic_make_request

Quoted 9 lines

The end result is that we miss the unplug completely. dm-crypt needs to unplug for sync bios. This patch also changes it to unplug whenever the queue is empty, which is far from ideal but better than missing the unplugs.

This doesn't completely fix io stalls I'm seeing with dm-crypt, but its my best guess. If it works, I'll break it up and submit for real to the dm people.

Quoted 4 lines

Out of curiousity, how are you measuring IO stalls? In the tests I'm doing, the worker processes output their progress and it should be at a steady rate. I considered a stall to be an excessive delay between updates which is a pretty indirect measure.

Quoted 7 lines

I just setup a crypto disk and did dd if=/dev/zero of=/mnt/foo bs=1M

If you watch vmstat 1, there's supposed to be a constant steam of IO to the disk. If a whole second goes by with zero IO, we're doing something wrong, I get a number of multi-second stalls where we are just waiting for IO to happen.

Ok, cool.

Quoted 4 lines

Most of the time I was able to catch a sysrq-w for it, someone was
waiting on a read to finish. It isn't completely clear to me if the
unplugging is working properly.

The machines I'm using are now tied up doing the same tests with dm-crypt and then I need to rerun with dm-crypt with the changed patches. It'll be early next week before the machines free up again for me to investigate your patch more but initial results look promising at least.

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
author
#12 lore raw

Re: [PATCH] make crypto unplug fix V3

In reply to Chris Mason
Quoted 52 lines · Chris Mason
Quoted 7 lines · Mel Gorman
Quoted 3 lines · Chris Mason

This is still likely to set your dm data on fire. It is only meant for testers that start with mkfs and don't have any valuable dm data.

The good news is that my room remains fire-free. Despite swap also running from dm-crypt, I had no corruption or instability issues.

Ok, definitely not so convincing I'd try and shove it into a late rc.

Quoted 41 lines

Here is an updated set of results for fake-gitk running.

X86

2.6.30-0000000-force-highorder           Elapsed:12:08.908    Failures:0
2.6.31-0000000-force-highorder           Elapsed:10:56.283    Failures:0
2.6.31-0000006-dm-crypt-unplug           Elapsed:11:51.653    Failures:0
2.6.31-0000012-pgalloc-2.6.30            Elapsed:12:26.587    Failures:0
2.6.31-0000123-congestion-both           Elapsed:10:55.298    Failures:0
2.6.31-0001234-kswapd-quick-recheck      Elapsed:18:01.523    Failures:0
2.6.31-0123456-dm-crypt-unplug           Elapsed:10:45.720    Failures:0
2.6.31-revert-8aa7e847                   Elapsed:15:08.020    Failures:0
2.6.32-rc6-0000000-force-highorder       Elapsed:16:20.765    Failures:4
2.6.32-rc6-0000006-dm-crypt-unplug       Elapsed:13:42.920    Failures:0
2.6.32-rc6-0000012-pgalloc-2.6.30        Elapsed:16:13.380    Failures:1
2.6.32-rc6-0000123-congestion-both       Elapsed:18:39.118    Failures:0
2.6.32-rc6-0001234-kswapd-quick-recheck  Elapsed:15:04.398    Failures:0
2.6.32-rc6-0123456-dm-crypt-unplug       Elapsed:12:50.438    Failures:0
2.6.32-rc6-revert-8aa7e847               Elapsed:20:50.888    Failures:0

X86-64

2.6.30-0000000-force-highorder           Elapsed:10:37.300    Failures:0
2.6.31-0000000-force-highorder           Elapsed:08:49.338    Failures:0
2.6.31-0000006-dm-crypt-unplug           Elapsed:09:37.840    Failures:0
2.6.31-0000012-pgalloc-2.6.30            Elapsed:15:49.690    Failures:0
2.6.31-0000123-congestion-both           Elapsed:09:18.790    Failures:0
2.6.31-0001234-kswapd-quick-recheck      Elapsed:08:39.268    Failures:0
2.6.31-0123456-dm-crypt-unplug           Elapsed:08:20.965    Failures:0
2.6.31-revert-8aa7e847                   Elapsed:08:07.457    Failures:0
2.6.32-rc6-0000000-force-highorder       Elapsed:18:29.103    Failures:1
2.6.32-rc6-0000006-dm-crypt-unplug       Elapsed:25:53.515    Failures:3
2.6.32-rc6-0000012-pgalloc-2.6.30        Elapsed:19:55.570    Failures:6
2.6.32-rc6-0000123-congestion-both       Elapsed:17:29.255    Failures:2
2.6.32-rc6-0001234-kswapd-quick-recheck  Elapsed:14:41.068    Failures:0
2.6.32-rc6-0123456-dm-crypt-unplug       Elapsed:15:48.028    Failures:1
2.6.32-rc6-revert-8aa7e847               Elapsed:14:48.647    Failures:0

The numbering in the kernel indicates what patches are applied. I tested the dm-crypt patch both in isolation and in combination with the patches in this series.

Quoted 8 lines
Quoted 4 lines

Basically, the dm-crypt-unplug makes a small difference in performance overall, mostly slight gains and losses. There was one massive regression with the dm-crypt patch applied to 2.6.32-rc6 but at the moment, I don't know what that is.

How consistent are your numbers between runs? I was trying to match this up with your last email and things were pretty different.

The figures from the first mail were based on kernels that were not instrumented. It so happened that this run was based on an instrumented kernel to get the congestion_wait figures so the results are different.

However, the results vary a lot. In some cases, it will spike just as 2.6.32-rc6-0000006-dm-crypt-unplug did. The reported figure is the average of 4 fake-gitk runs. I don't have the standard deviation handy but it's high.

Quoted 29 lines
Quoted 24 lines

In general, the patch reduces the amount of time direct reclaimers are spending on congestion_wait.

Quoted 10 lines

It includes my patch from last night, along with changes to force dm to unplug when its IO queues empty.

The problem goes like this:

Process: submit read bio
dm: put bio onto work queue
process: unplug
dm: work queue finds bio, does a generic_make_request

Quoted 9 lines

The end result is that we miss the unplug completely. dm-crypt needs to unplug for sync bios. This patch also changes it to unplug whenever the queue is empty, which is far from ideal but better than missing the unplugs.

This doesn't completely fix io stalls I'm seeing with dm-crypt, but its my best guess. If it works, I'll break it up and submit for real to the dm people.

Quoted 4 lines

Out of curiousity, how are you measuring IO stalls? In the tests I'm doing, the worker processes output their progress and it should be at a steady rate. I considered a stall to be an excessive delay between updates which is a pretty indirect measure.

Quoted 7 lines

I just setup a crypto disk and did dd if=/dev/zero of=/mnt/foo bs=1M

If you watch vmstat 1, there's supposed to be a constant steam of IO to the disk. If a whole second goes by with zero IO, we're doing something wrong, I get a number of multi-second stalls where we are just waiting for IO to happen.

Ok, cool.

Quoted 4 lines

Most of the time I was able to catch a sysrq-w for it, someone was
waiting on a read to finish. It isn't completely clear to me if the
unplugging is working properly.

The machines I'm using are now tied up doing the same tests with dm-crypt and then I need to rerun with dm-crypt with the changed patches. It'll be early next week before the machines free up again for me to investigate your patch more but initial results look promising at least.

Signature · 9 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
author
#13 lore raw

Re: [PATCH] make crypto unplug fix V3

In reply to Chris Mason
Quoted 3 lines · Chris Mason

This is still likely to set your dm data on fire. It is only meant for testers that start with mkfs and don't have any valuable dm data.

The good news is that my room remains fire-free. Despite swap also running from dm-crypt, I had no corruption or instability issues.

Here is an updated set of results for fake-gitk running.

X86

2.6.30-0000000-force-highorder           Elapsed:12:08.908    Failures:0
2.6.31-0000000-force-highorder           Elapsed:10:56.283    Failures:0
2.6.31-0000006-dm-crypt-unplug           Elapsed:11:51.653    Failures:0
2.6.31-0000012-pgalloc-2.6.30            Elapsed:12:26.587    Failures:0
2.6.31-0000123-congestion-both           Elapsed:10:55.298    Failures:0
2.6.31-0001234-kswapd-quick-recheck      Elapsed:18:01.523    Failures:0
2.6.31-0123456-dm-crypt-unplug           Elapsed:10:45.720    Failures:0
2.6.31-revert-8aa7e847                   Elapsed:15:08.020    Failures:0
2.6.32-rc6-0000000-force-highorder       Elapsed:16:20.765    Failures:4
2.6.32-rc6-0000006-dm-crypt-unplug       Elapsed:13:42.920    Failures:0
2.6.32-rc6-0000012-pgalloc-2.6.30        Elapsed:16:13.380    Failures:1
2.6.32-rc6-0000123-congestion-both       Elapsed:18:39.118    Failures:0
2.6.32-rc6-0001234-kswapd-quick-recheck  Elapsed:15:04.398    Failures:0
2.6.32-rc6-0123456-dm-crypt-unplug       Elapsed:12:50.438    Failures:0
2.6.32-rc6-revert-8aa7e847               Elapsed:20:50.888    Failures:0

X86-64

2.6.30-0000000-force-highorder           Elapsed:10:37.300    Failures:0
2.6.31-0000000-force-highorder           Elapsed:08:49.338    Failures:0
2.6.31-0000006-dm-crypt-unplug           Elapsed:09:37.840    Failures:0
2.6.31-0000012-pgalloc-2.6.30            Elapsed:15:49.690    Failures:0
2.6.31-0000123-congestion-both           Elapsed:09:18.790    Failures:0
2.6.31-0001234-kswapd-quick-recheck      Elapsed:08:39.268    Failures:0
2.6.31-0123456-dm-crypt-unplug           Elapsed:08:20.965    Failures:0
2.6.31-revert-8aa7e847                   Elapsed:08:07.457    Failures:0
2.6.32-rc6-0000000-force-highorder       Elapsed:18:29.103    Failures:1
2.6.32-rc6-0000006-dm-crypt-unplug       Elapsed:25:53.515    Failures:3
2.6.32-rc6-0000012-pgalloc-2.6.30        Elapsed:19:55.570    Failures:6
2.6.32-rc6-0000123-congestion-both       Elapsed:17:29.255    Failures:2
2.6.32-rc6-0001234-kswapd-quick-recheck  Elapsed:14:41.068    Failures:0
2.6.32-rc6-0123456-dm-crypt-unplug       Elapsed:15:48.028    Failures:1
2.6.32-rc6-revert-8aa7e847               Elapsed:14:48.647    Failures:0

The numbering in the kernel indicates what patches are applied. I tested the dm-crypt patch both in isolation and in combination with the patches in this series.

Basically, the dm-crypt-unplug makes a small difference in performance overall, mostly slight gains and losses. There was one massive regression with the dm-crypt patch applied to 2.6.32-rc6 but at the moment, I don't know what that is.

In general, the patch reduces the amount of time direct reclaimers are spending on congestion_wait.

Quoted 10 lines

It includes my patch from last night, along with changes to force dm to unplug when its IO queues empty.

The problem goes like this:

Process: submit read bio
dm: put bio onto work queue
process: unplug
dm: work queue finds bio, does a generic_make_request

Quoted 9 lines

The end result is that we miss the unplug completely. dm-crypt needs to unplug for sync bios. This patch also changes it to unplug whenever the queue is empty, which is far from ideal but better than missing the unplugs.

This doesn't completely fix io stalls I'm seeing with dm-crypt, but its my best guess. If it works, I'll break it up and submit for real to the dm people.

Out of curiousity, how are you measuring IO stalls? In the tests I'm doing, the worker processes output their progress and it should be at a steady rate. I considered a stall to be an excessive delay between updates which is a pretty indirect measure.

Quoted 108 lines
(quoted diff) +0 -1
-chris
drivers/md/dm-crypt.c +42 -3
index ed10381..729ae01 100644
@@ -94,8 +94,12 @@ struct crypt_config {
 	struct bio_set *bs;
 
 	struct workqueue_struct *io_queue;
+	struct workqueue_struct *async_io_queue;
 	struct workqueue_struct *crypt_queue;
 
+	atomic_t sync_bios_in_queue;
+	atomic_t async_bios_in_queue;
+
 	/*
 	 * crypto related data
 	 */
@@ -679,11 +683,29 @@ static void kcryptd_io_write(struct dm_crypt_io *io)
 static void kcryptd_io(struct work_struct *work)
 {
 	struct dm_crypt_io *io = container_of(work, struct dm_crypt_io, work);
+	struct crypt_config *cc = io->target->private;
+	int zero_sync = 0;
+	int zero_async = 0;
+	int was_sync = 0;
+
+	if (io->base_bio->bi_rw & (1 << BIO_RW_SYNCIO)) {
+		zero_sync = atomic_dec_and_test(&cc->sync_bios_in_queue);
+		was_sync = 1;
+	} else
+		zero_async = atomic_dec_and_test(&cc->async_bios_in_queue);
 
 	if (bio_data_dir(io->base_bio) == READ)
 		kcryptd_io_read(io);
 	else
 		kcryptd_io_write(io);
+
+	if ((was_sync && zero_sync) ||
+	    (!was_sync && zero_async &&
+	     atomic_read(&cc->sync_bios_in_queue) == 0)) {
+		struct backing_dev_info *bdi;
+		bdi = blk_get_backing_dev_info(io->base_bio->bi_bdev);
+		blk_run_backing_dev(bdi, NULL);
+	}
 }
 
 static void kcryptd_queue_io(struct dm_crypt_io *io)
@@ -691,7 +713,13 @@ static void kcryptd_queue_io(struct dm_crypt_io *io)
 	struct crypt_config *cc = io->target->private;
 
 	INIT_WORK(&io->work, kcryptd_io);
-	queue_work(cc->io_queue, &io->work);
+	if (io->base_bio->bi_rw & (1 << BIO_RW_SYNCIO)) {
+		atomic_inc(&cc->sync_bios_in_queue);
+		queue_work(cc->io_queue, &io->work);
+	} else {
+		atomic_inc(&cc->async_bios_in_queue);
+		queue_work(cc->async_io_queue, &io->work);
+	}
 }
 
 static void kcryptd_crypt_write_io_submit(struct dm_crypt_io *io,
@@ -759,8 +787,7 @@ static void kcryptd_crypt_write_convert(struct dm_crypt_io *io)
 
 		/* Encryption was already finished, submit io now */
 		if (crypt_finished) {
-			kcryptd_crypt_write_io_submit(io, r, 0);
-
+			kcryptd_crypt_write_io_submit(io, r, 1);
 			/*
 			 * If there was an error, do not try next fragments.
 			 * For async, error is processed in async handler.
@@ -1120,6 +1147,15 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
 	} else
 		cc->iv_mode = NULL;
 
+	atomic_set(&cc->sync_bios_in_queue, 0);
+	atomic_set(&cc->async_bios_in_queue, 0);
+
+	cc->async_io_queue = create_singlethread_workqueue("kcryptd_async_io");
+	if (!cc->async_io_queue) {
+		ti->error = "Couldn't create kcryptd io queue";
+		goto bad_async_io_queue;
+	}
+
 	cc->io_queue = create_singlethread_workqueue("kcryptd_io");
 	if (!cc->io_queue) {
 		ti->error = "Couldn't create kcryptd io queue";
@@ -1139,6 +1175,8 @@ static int crypt_ctr(struct dm_target *ti, unsigned int argc, char **argv)
 bad_crypt_queue:
 	destroy_workqueue(cc->io_queue);
 bad_io_queue:
+	destroy_workqueue(cc->async_io_queue);
+bad_async_io_queue:
 	kfree(cc->iv_mode);
 bad_ivmode_string:
 	dm_put_device(ti, cc->dev);
@@ -1166,6 +1204,7 @@ static void crypt_dtr(struct dm_target *ti)
 	struct crypt_config *cc = (struct crypt_config *) ti->private;
 
 	destroy_workqueue(cc->io_queue);
+	destroy_workqueue(cc->async_io_queue);
 	destroy_workqueue(cc->crypt_queue);
 
 	if (cc->req)
Signature · 9 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#14 lore raw
In reply to Chris Mason
Quoted 9 lines · Chris Mason

On Thu, Nov 12, 2009 at 05:00:05PM -0500, Chris Mason wrote:

[ ...]

Quoted 5 lines

The punch line is that the btrfs guy thinks we can solve all of this with just one more thread. If we change dm-crypt to have a thread dedicated to sync IO and a thread dedicated to async IO the system should smooth out.

Please, can you cc DM maintainers with these kind of patches? dm-devel list at least.

Note that the crypt requests can be already processed synchronously or asynchronously, depending on used crypto module (async it is in the case of some hw acceleration).

Adding another queue make the situation more complicated and because the crypt requests can be queued in crypto layer I am not sure that this solution will help in this situation at all.
(Try to run that with AES-NI acceleration for example.)

Milan
--
[email protected]

#15 lore raw
Quoted 6 lines · Milan Broz
Quoted 5 lines · Chris Mason

On Thu, Nov 12, 2009 at 05:00:05PM -0500, Chris Mason wrote:

[ ...]

Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

The punch line is that the btrfs guy thinks we can solve all of this with just one more thread. If we change dm-crypt to have a thread dedicated to sync IO and a thread dedicated to async IO the system should smooth out.

Please, can you cc DM maintainers with these kind of patches? dm-devel list at least.

Well, my current patch is a hack. If I had come up with a proven theory (hopefully Mel can prove it ;), it definitely would have gone through the dm-devel lists.

Quoted 7 lines

Note that the crypt requests can be already processed synchronously or asynchronously, depending on used crypto module (async it is in the case of some hw acceleration).

Adding another queue make the situation more complicated and because the crypt requests can be queued in crypto layer I am not sure that this solution will help in this situation at all.
(Try to run that with AES-NI acceleration for example.)

The problem is that async threads still imply a kind of ordering. If there's a fifo serviced by one thread or 10, the latency ramifications are very similar for a new entry on the list. We have to wait for a large portion of the low-prio items in order to service a high prio item.

With a queue dedicated to sync requests and one dedicated to async, you'll get better read latencies. Btrfs has a similar problem around the crc helper threads and it ends up solving things with two different lists (high and low prio) processed by one thread.

-chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#16 lore raw
Quoted 12 lines · Chris Mason
Quoted 11 lines · Milan Broz
Quoted 9 lines · Chris Mason

On Thu, Nov 12, 2009 at 05:00:05PM -0500, Chris Mason wrote:

[ ...]

Quoted 5 lines

The punch line is that the btrfs guy thinks we can solve all of this with just one more thread. If we change dm-crypt to have a thread dedicated to sync IO and a thread dedicated to async IO the system should smooth out.

Quoted 7 lines
Quoted 2 lines

Please, can you cc DM maintainers with these kind of patches? dm-devel list at least.

Well, my current patch is a hack. If I had come up with a proven theory (hopefully Mel can prove it ;), it definitely would have gone through the dm-devel lists.

I can't prove it for sure but the workload might not be targetted enough to show better or worse read latencies.

I adjusted the workload to run fake-gitk multiple times to get a better sense of the deviation between runs

On X86-64, the timings were

2.6.30-0000000-force-highorder                Elapsed:10:52.218(stddev:008.085)   Failures:0
2.6.31-0000000-force-highorder                Elapsed:11:32.258(stddev:130.779)   Failures:0
2.6.31-0012345-kswapd-stay-awake-when-min     Elapsed:09:34.662(stddev:022.239)   Failures:0
2.6.31-0123456-dm-crypt-unplug                Elapsed:10:28.718(stddev:060.897)   Failures:0
2.6.32-rc6-0000000-force-highorder            Elapsed:27:53.686(stddev:207.508)   Failures:37
2.6.32-rc6-0012345-kswapd-stay-awake-when-min Elapsed:27:26.735(stddev:221.214)   Failures:6
2.6.32-rc6-0123456-dm-crypt-unplug            Elapsed:27:35.462(stddev:205.017)   Failures:4

On X86, they were

2.6.30-0000000-force-highorder                Elapsed:13:36.768(stddev:019.514)   Failures:0
2.6.31-0000000-force-highorder                Elapsed:16:27.922(stddev:134.839)   Failures:0
2.6.31-0000006-dm-crypt-unplug                Elapsed:15:47.160(stddev:183.488)   Failures:0
2.6.31-0012345-kswapd-stay-awake-when-min     Elapsed:18:32.458(stddev:182.164)   Failures:0
2.6.31-0123456-dm-crypt-unplug                Elapsed:17:07.482(stddev:210.404)   Failures:0
2.6.32-rc6-0000000-force-highorder            Elapsed:26:08.763(stddev:123.926)   Failures:4
2.6.32-rc6-0000006-dm-crypt-unplug            Elapsed:17:57.550(stddev:254.412)   Failures:1
2.6.32-rc6-0012345-kswapd-stay-awake-when-min Elapsed:25:03.435(stddev:234.685)   Failures:1
2.6.32-rc6-0123456-dm-crypt-unplug            Elapsed:25:21.382(stddev:211.252)   Failures:0

(I forgot to queue up the dm-crypt patches on their own for X86-64 which is why the results are missing).

While the dm-crypt patch shows small differences, they are well within the noise for each run of fake-gitk so I can't draw any major conclusion from it.

On X86 for 2.6.31, roughly the same amount of time is spent in congestion_wait() with or without the patch. On 2.6.32-rc6, the time kswapd spends congestioned on the ASYNC queue is reduced by about 20% both when compared against mainline and compared against the other patches in the series applied. There is very little difference to the congestion on the SYNC queue.

On X86-64 for 2.6.31, the story is slightly different. I don't think it's an architecture thing because the X86-64 machine has twice as many cores as the X86 test machine. Here, congestion_wait() spent on the ASYNC queue remains roughly the same but the time spent on the SYNC queue for direct reclaim is reduced by almost a third. Against 2.6.32-rc6, there was very little difference.

Again, it's hard to draw solid conclusions from this. I know from other testing that the low_latency tunable for the IO scheduler is an important factor for the performance of this test on 2.6.32-rc6 so if disabled, it mgiht show a clearer picture, but right now I can't say for sure it's an improvement.

Quoted 21 lines
Quoted 7 lines

Note that the crypt requests can be already processed synchronously or asynchronously, depending on used crypto module (async it is in the case of some hw acceleration).

Adding another queue make the situation more complicated and because the crypt requests can be queued in crypto layer I am not sure that this solution will help in this situation at all.
(Try to run that with AES-NI acceleration for example.)

The problem is that async threads still imply a kind of ordering. If there's a fifo serviced by one thread or 10, the latency ramifications are very similar for a new entry on the list. We have to wait for a large portion of the low-prio items in order to service a high prio item.

With a queue dedicated to sync requests and one dedicated to async, you'll get better read latencies. Btrfs has a similar problem around the crc helper threads and it ends up solving things with two different lists (high and low prio) processed by one thread.

-chris

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
author
#17 lore raw
In reply to Chris Mason
Quoted 12 lines · Chris Mason
Quoted 11 lines · Milan Broz
Quoted 9 lines · Chris Mason

On Thu, Nov 12, 2009 at 05:00:05PM -0500, Chris Mason wrote:

[ ...]

Quoted 5 lines

The punch line is that the btrfs guy thinks we can solve all of this with just one more thread. If we change dm-crypt to have a thread dedicated to sync IO and a thread dedicated to async IO the system should smooth out.

Quoted 7 lines
Quoted 2 lines

Please, can you cc DM maintainers with these kind of patches? dm-devel list at least.

Well, my current patch is a hack. If I had come up with a proven theory (hopefully Mel can prove it ;), it definitely would have gone through the dm-devel lists.

I can't prove it for sure but the workload might not be targetted enough to show better or worse read latencies.

I adjusted the workload to run fake-gitk multiple times to get a better sense of the deviation between runs

On X86-64, the timings were

2.6.30-0000000-force-highorder                Elapsed:10:52.218(stddev:008.085)   Failures:0
2.6.31-0000000-force-highorder                Elapsed:11:32.258(stddev:130.779)   Failures:0
2.6.31-0012345-kswapd-stay-awake-when-min     Elapsed:09:34.662(stddev:022.239)   Failures:0
2.6.31-0123456-dm-crypt-unplug                Elapsed:10:28.718(stddev:060.897)   Failures:0
2.6.32-rc6-0000000-force-highorder            Elapsed:27:53.686(stddev:207.508)   Failures:37
2.6.32-rc6-0012345-kswapd-stay-awake-when-min Elapsed:27:26.735(stddev:221.214)   Failures:6
2.6.32-rc6-0123456-dm-crypt-unplug            Elapsed:27:35.462(stddev:205.017)   Failures:4

On X86, they were

2.6.30-0000000-force-highorder                Elapsed:13:36.768(stddev:019.514)   Failures:0
2.6.31-0000000-force-highorder                Elapsed:16:27.922(stddev:134.839)   Failures:0
2.6.31-0000006-dm-crypt-unplug                Elapsed:15:47.160(stddev:183.488)   Failures:0
2.6.31-0012345-kswapd-stay-awake-when-min     Elapsed:18:32.458(stddev:182.164)   Failures:0
2.6.31-0123456-dm-crypt-unplug                Elapsed:17:07.482(stddev:210.404)   Failures:0
2.6.32-rc6-0000000-force-highorder            Elapsed:26:08.763(stddev:123.926)   Failures:4
2.6.32-rc6-0000006-dm-crypt-unplug            Elapsed:17:57.550(stddev:254.412)   Failures:1
2.6.32-rc6-0012345-kswapd-stay-awake-when-min Elapsed:25:03.435(stddev:234.685)   Failures:1
2.6.32-rc6-0123456-dm-crypt-unplug            Elapsed:25:21.382(stddev:211.252)   Failures:0

(I forgot to queue up the dm-crypt patches on their own for X86-64 which is why the results are missing).

While the dm-crypt patch shows small differences, they are well within the noise for each run of fake-gitk so I can't draw any major conclusion from it.

On X86 for 2.6.31, roughly the same amount of time is spent in congestion_wait() with or without the patch. On 2.6.32-rc6, the time kswapd spends congestioned on the ASYNC queue is reduced by about 20% both when compared against mainline and compared against the other patches in the series applied. There is very little difference to the congestion on the SYNC queue.

On X86-64 for 2.6.31, the story is slightly different. I don't think it's an architecture thing because the X86-64 machine has twice as many cores as the X86 test machine. Here, congestion_wait() spent on the ASYNC queue remains roughly the same but the time spent on the SYNC queue for direct reclaim is reduced by almost a third. Against 2.6.32-rc6, there was very little difference.

Again, it's hard to draw solid conclusions from this. I know from other testing that the low_latency tunable for the IO scheduler is an important factor for the performance of this test on 2.6.32-rc6 so if disabled, it mgiht show a clearer picture, but right now I can't say for sure it's an improvement.

Quoted 21 lines
Quoted 7 lines

Note that the crypt requests can be already processed synchronously or asynchronously, depending on used crypto module (async it is in the case of some hw acceleration).

Adding another queue make the situation more complicated and because the crypt requests can be queued in crypto layer I am not sure that this solution will help in this situation at all.
(Try to run that with AES-NI acceleration for example.)

The problem is that async threads still imply a kind of ordering. If there's a fifo serviced by one thread or 10, the latency ramifications are very similar for a new entry on the list. We have to wait for a large portion of the low-prio items in order to service a high prio item.

With a queue dedicated to sync requests and one dedicated to async, you'll get better read latencies. Btrfs has a similar problem around the crc helper threads and it ends up solving things with two different lists (high and low prio) processed by one thread.

-chris

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#18 lore raw
In reply to Chris Mason
Quoted 12 lines · Chris Mason
Quoted 11 lines · Milan Broz
Quoted 9 lines · Chris Mason

On Thu, Nov 12, 2009 at 05:00:05PM -0500, Chris Mason wrote:

[ ...]

Quoted 5 lines

The punch line is that the btrfs guy thinks we can solve all of this with just one more thread. If we change dm-crypt to have a thread dedicated to sync IO and a thread dedicated to async IO the system should smooth out.

Quoted 7 lines
Quoted 2 lines

Please, can you cc DM maintainers with these kind of patches? dm-devel list at least.

Well, my current patch is a hack. If I had come up with a proven theory (hopefully Mel can prove it ;), it definitely would have gone through the dm-devel lists.

I can't prove it for sure but the workload might not be targetted enough to show better or worse read latencies.

I adjusted the workload to run fake-gitk multiple times to get a better sense of the deviation between runs

On X86-64, the timings were

2.6.30-0000000-force-highorder                Elapsed:10:52.218(stddev:008.085)   Failures:0
2.6.31-0000000-force-highorder                Elapsed:11:32.258(stddev:130.779)   Failures:0
2.6.31-0012345-kswapd-stay-awake-when-min     Elapsed:09:34.662(stddev:022.239)   Failures:0
2.6.31-0123456-dm-crypt-unplug                Elapsed:10:28.718(stddev:060.897)   Failures:0
2.6.32-rc6-0000000-force-highorder            Elapsed:27:53.686(stddev:207.508)   Failures:37
2.6.32-rc6-0012345-kswapd-stay-awake-when-min Elapsed:27:26.735(stddev:221.214)   Failures:6
2.6.32-rc6-0123456-dm-crypt-unplug            Elapsed:27:35.462(stddev:205.017)   Failures:4

On X86, they were

2.6.30-0000000-force-highorder                Elapsed:13:36.768(stddev:019.514)   Failures:0
2.6.31-0000000-force-highorder                Elapsed:16:27.922(stddev:134.839)   Failures:0
2.6.31-0000006-dm-crypt-unplug                Elapsed:15:47.160(stddev:183.488)   Failures:0
2.6.31-0012345-kswapd-stay-awake-when-min     Elapsed:18:32.458(stddev:182.164)   Failures:0
2.6.31-0123456-dm-crypt-unplug                Elapsed:17:07.482(stddev:210.404)   Failures:0
2.6.32-rc6-0000000-force-highorder            Elapsed:26:08.763(stddev:123.926)   Failures:4
2.6.32-rc6-0000006-dm-crypt-unplug            Elapsed:17:57.550(stddev:254.412)   Failures:1
2.6.32-rc6-0012345-kswapd-stay-awake-when-min Elapsed:25:03.435(stddev:234.685)   Failures:1
2.6.32-rc6-0123456-dm-crypt-unplug            Elapsed:25:21.382(stddev:211.252)   Failures:0

(I forgot to queue up the dm-crypt patches on their own for X86-64 which is why the results are missing).

While the dm-crypt patch shows small differences, they are well within the noise for each run of fake-gitk so I can't draw any major conclusion from it.

On X86 for 2.6.31, roughly the same amount of time is spent in congestion_wait() with or without the patch. On 2.6.32-rc6, the time kswapd spends congestioned on the ASYNC queue is reduced by about 20% both when compared against mainline and compared against the other patches in the series applied. There is very little difference to the congestion on the SYNC queue.

On X86-64 for 2.6.31, the story is slightly different. I don't think it's an architecture thing because the X86-64 machine has twice as many cores as the X86 test machine. Here, congestion_wait() spent on the ASYNC queue remains roughly the same but the time spent on the SYNC queue for direct reclaim is reduced by almost a third. Against 2.6.32-rc6, there was very little difference.

Again, it's hard to draw solid conclusions from this. I know from other testing that the low_latency tunable for the IO scheduler is an important factor for the performance of this test on 2.6.32-rc6 so if disabled, it mgiht show a clearer picture, but right now I can't say for sure it's an improvement.

Quoted 21 lines
Quoted 7 lines

Note that the crypt requests can be already processed synchronously or asynchronously, depending on used crypto module (async it is in the case of some hw acceleration).

Adding another queue make the situation more complicated and because the crypt requests can be queued in crypto layer I am not sure that this solution will help in this situation at all.
(Try to run that with AES-NI acceleration for example.)

The problem is that async threads still imply a kind of ordering. If there's a fifo serviced by one thread or 10, the latency ramifications are very similar for a new entry on the list. We have to wait for a large portion of the low-prio items in order to service a high prio item.

With a queue dedicated to sync requests and one dedicated to async, you'll get better read latencies. Btrfs has a similar problem around the crc helper threads and it ends up solving things with two different lists (high and low prio) processed by one thread.

-chris

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#19 lore raw
In reply to Chris Mason
Quoted 9 lines · Chris Mason

On Thu, Nov 12, 2009 at 05:00:05PM -0500, Chris Mason wrote:

[ ...]

Quoted 5 lines

The punch line is that the btrfs guy thinks we can solve all of this with just one more thread. If we change dm-crypt to have a thread dedicated to sync IO and a thread dedicated to async IO the system should smooth out.

Please, can you cc DM maintainers with these kind of patches? dm-devel list at least.

Note that the crypt requests can be already processed synchronously or asynchronously, depending on used crypto module (async it is in the case of some hw acceleration).

Adding another queue make the situation more complicated and because the crypt requests can be queued in crypto layer I am not sure that this solution will help in this situation at all.
(Try to run that with AES-NI acceleration for example.)

Milan
--
[email protected]

#20 lore raw
In reply to Chris Mason
Quoted 9 lines · Chris Mason

On Thu, Nov 12, 2009 at 05:00:05PM -0500, Chris Mason wrote:

[ ...]

Quoted 5 lines

The punch line is that the btrfs guy thinks we can solve all of this with just one more thread. If we change dm-crypt to have a thread dedicated to sync IO and a thread dedicated to async IO the system should smooth out.

Please, can you cc DM maintainers with these kind of patches? dm-devel list at least.

Note that the crypt requests can be already processed synchronously or asynchronously, depending on used crypto module (async it is in the case of some hw acceleration).

Adding another queue make the situation more complicated and because the crypt requests can be queued in crypto layer I am not sure that this solution will help in this situation at all.
(Try to run that with AES-NI acceleration for example.)

Milan
--
[email protected]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#21 lore raw
In reply to Chris Mason
Quoted 19 lines · Chris Mason
Quoted 18 lines · Chris Mason
Quoted 16 lines · Mel Gorman

Sorry for the long delay in posting another version. Testing is extremely time-consuming and I wasn't getting to work on this as much as I'd have liked.

Changelog since V2
o Dropped the kswapd-quickly-notice-high-order patch. In more detailed testing, it made latencies even worse as kswapd slept more on high-order congestion causing order-0 direct reclaims.
o Added changes to how congestion_wait() works
o Added a number of new patches altering the behaviour of reclaim

Since 2.6.31-rc1, there have been an increasing number of GFP_ATOMIC failures. A significant number of these have been high-order GFP_ATOMIC failures and while they are generally brushed away, there has been a large increase in them recently and there are a number of possible areas the problem could be in - core vm, page writeback and a specific driver. The bugs affected by this that I am aware of are;

Quoted 8 lines
Quoted 5 lines

Thanks for all the time you've spent on this one. Let me start with some more questions about the workload ;)

So the workload is gitk reading a git repo and a program reading data over the network. Which part of the workload writes to disk?

Sorry for the self reply, I started digging through your data (man, that's a lot of data ;).

Yeah, sorry about that. Because I lacked a credible explanation as to why waiting on sync really made such a difference, I had little choice but to punt everything I had for people to dig through.

To be clear, I'm not actually running gitk. The fake-gitk is reading the commits into memory and building a tree in a similar fashion to what gitk does. I didn't want to use gitk itself because there wasn't a way of measuring whether it was stalling or just other than looking at it and making a guess.

Quoted 20 lines

I took another tour through dm-crypt and
things make more sense now.

dm-crypt has two different single threaded workqueues for each dm-crypt device. The first one is meant to deal with the actual encryption and decryption, and the second one is meant to do the IO.

So the path for a write looks something like this:

filesystem -> crypt thread -> encrypt the data -> io thread -> disk

And the path for read looks something like this:

filesystem -> io thread -> disk -> crypt thread -> decrypt data -> FS

One thread does encryption and one thread does IO, and these threads are shared for reads and writes. The end result is that all of the sync reads get stuck behind any async write congestion and all of the async writes get stuck behind any sync read congestion.

Quoted 9 lines

It's almost like you need to check for both sync and async congestion before you have any hopes of a new IO making progress.

The confusing part is that dm hasn't gotten any worse in this regard since 2.6.30 but the workload here is generating more sync reads (hopefully from gitk and swapin) than async writes (from the low bandwidth rsync). So in general if you were to change mm/*.c wait for sync congestion instead of async, things should appear better.

Thanks very much for that explanation. It makes a lot of sense and explains why waiting on sync-congestion made such a difference on the test setup.

Quoted 5 lines

The punch line is that the btrfs guy thinks we can solve all of this with just one more thread. If we change dm-crypt to have a thread dedicated to sync IO and a thread dedicated to async IO the system should smooth out.

I see you have posted another patch so I'll test that out first before looking into that.

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
author
#22 lore raw
In reply to Chris Mason
Quoted 19 lines · Chris Mason
Quoted 18 lines · Chris Mason
Quoted 16 lines · Mel Gorman

Sorry for the long delay in posting another version. Testing is extremely time-consuming and I wasn't getting to work on this as much as I'd have liked.

Changelog since V2
o Dropped the kswapd-quickly-notice-high-order patch. In more detailed testing, it made latencies even worse as kswapd slept more on high-order congestion causing order-0 direct reclaims.
o Added changes to how congestion_wait() works
o Added a number of new patches altering the behaviour of reclaim

Since 2.6.31-rc1, there have been an increasing number of GFP_ATOMIC failures. A significant number of these have been high-order GFP_ATOMIC failures and while they are generally brushed away, there has been a large increase in them recently and there are a number of possible areas the problem could be in - core vm, page writeback and a specific driver. The bugs affected by this that I am aware of are;

Quoted 8 lines
Quoted 5 lines

Thanks for all the time you've spent on this one. Let me start with some more questions about the workload ;)

So the workload is gitk reading a git repo and a program reading data over the network. Which part of the workload writes to disk?

Sorry for the self reply, I started digging through your data (man, that's a lot of data ;).

Yeah, sorry about that. Because I lacked a credible explanation as to why waiting on sync really made such a difference, I had little choice but to punt everything I had for people to dig through.

To be clear, I'm not actually running gitk. The fake-gitk is reading the commits into memory and building a tree in a similar fashion to what gitk does. I didn't want to use gitk itself because there wasn't a way of measuring whether it was stalling or just other than looking at it and making a guess.

Quoted 20 lines

I took another tour through dm-crypt and
things make more sense now.

dm-crypt has two different single threaded workqueues for each dm-crypt device. The first one is meant to deal with the actual encryption and decryption, and the second one is meant to do the IO.

So the path for a write looks something like this:

filesystem -> crypt thread -> encrypt the data -> io thread -> disk

And the path for read looks something like this:

filesystem -> io thread -> disk -> crypt thread -> decrypt data -> FS

One thread does encryption and one thread does IO, and these threads are shared for reads and writes. The end result is that all of the sync reads get stuck behind any async write congestion and all of the async writes get stuck behind any sync read congestion.

Quoted 9 lines

It's almost like you need to check for both sync and async congestion before you have any hopes of a new IO making progress.

The confusing part is that dm hasn't gotten any worse in this regard since 2.6.30 but the workload here is generating more sync reads (hopefully from gitk and swapin) than async writes (from the low bandwidth rsync). So in general if you were to change mm/*.c wait for sync congestion instead of async, things should appear better.

Thanks very much for that explanation. It makes a lot of sense and explains why waiting on sync-congestion made such a difference on the test setup.

Quoted 5 lines

The punch line is that the btrfs guy thinks we can solve all of this with just one more thread. If we change dm-crypt to have a thread dedicated to sync IO and a thread dedicated to async IO the system should smooth out.

I see you have posted another patch so I'll test that out first before looking into that.

Signature · 9 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>