dm-crypt flush-to-disk freezes
Hi all,
now I have been hit with a flush-to-disk problem.
The set-up is a dm-crypt partition with a Windows XP VM and the current VMplayer, all with kernel 2.6.34.4 from kernel.org and current vmware-tools in the VM.
The XP in the VM (which I use when I cannot avoid working with Word) frequently hangs for single digit seconds, on what seems to be flushes-to-disk.
Moving the VM to an unencrypted partition cures the problem, but as I work on confidential stuff, I would rather have it encrypted.
What is the current take on this problem? Any fix
known or expected for the near future?
Thanks for any insights.
Arno
Signature · 7 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: [email protected] GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier
Quoted 2 lines · Arno Wagner
What is the current take on this problem? Any fix
known or expected for the near future?
I don't know, and I'm not a VM proff either. That said, does it improve things when you write e.g. the following in your sysctl.conf:
vm.dirty_ratio = 10
vm.dirty_background_ratio = 5
Or you could go even lower using
vm.dirty_background_bytes = 134217728
vm.dirty_bytes = 268435456
(but then without setting vm.dirty_ratio and vm.dirty_background). It causes pdflush to write data more frequently and in smaller amounts, thus reducing blocking via disk i/o.
I've had io-locks in 2.6.32, but they seem to have disappeared in 2.6.34 (gentoo hardened-r1).
At least the frequency is drastically less,as previously under heavy IO I would get lock ups weekly, and now under the same load, my kernel hasn't been locking up for two to three weeks. Of course, this is just anecdotal evidence, and not based on any change-logs...
Quoted 20 lines · On Wed, 18 Aug 2010 10:31:00 +0200 Heinz Diehl <[email protected]> wrote:
Quoted 2 lines · Arno Wagner
What is the current take on this problem? Any fix
known or expected for the near future?I don't know, and I'm not a VM proff either. That said, does it improve things when you write e.g. the following in your sysctl.conf:
vm.dirty_ratio = 10
vm.dirty_background_ratio = 5Or you could go even lower using
vm.dirty_background_bytes = 134217728
vm.dirty_bytes = 268435456(but then without setting vm.dirty_ratio and vm.dirty_background). It causes pdflush to write data more frequently and in smaller amounts, thus reducing blocking via disk i/o.
Quoted 3 lines · Arno Wagner
The set-up is a dm-crypt partition with a Windows XP VM and the current VMplayer, all with kernel 2.6.34.4 from kernel.org and current vmware-tools in the VM.
Can you be more specific?
You have direct partition with LUKS and this is mapped to vmware directly as disk, right?
No fs in the middle?
Are there other encrypted disks on the same underlying device?
Underlying disk is not SSD, right?
What is the preemption type set in Linux kernel (Processor type/Preemption model in kernel config)?
What is the IO scheduler for the underlying device? (You can try deadline or cfq with some tuning - this is possible to tune online).
Probably tuning fs inside vm can help too. If it is massive syncing with lot of data waiting for completion, it is not good.
btw I hope that in 2.6.36 we switch to global workqueues per cpu for dm-crypt, unfortunately Alasdair is still waiting with committing it (patches are ready and I am using it myself). (It can help or it can make it worse here. It needs testing.)
Milan
Quoted 4 lines · Milan Broz
btw I hope that in 2.6.36 we switch to global workqueues per cpu for dm-crypt, unfortunately Alasdair is still waiting with committing it (patches are ready and I am using it myself). (It can help or it can make it worse here. It needs testing.)
Where can I get these patches?
Quoted 8 lines · Heinz Diehl
Quoted 4 lines · Milan Broz
btw I hope that in 2.6.36 we switch to global workqueues per cpu for dm-crypt, unfortunately Alasdair is still waiting with committing it (patches are ready and I am using it myself). (It can help or it can make it worse here. It needs testing.)
Where can I get these patches?
Not yet in upstream dm tree still...
So my working version is temporarily here
http://mbroz.fedorapeople.org/dm-crypt/
It basically patch from Andi Kleen with some minor fixes, split into 4 parts for easy review.
(You have to apply _always_ all 4 patches.)
Version for 2.6.34/35 is split into one file (actually I am using this version on my stable machine now).
Probably this is what you want.
Please note one thing - it creates threads per CPU, but work submitted by one cpu should never be processed by other thread. So it helps if you have parallel application but will not help too much if all work is committed from one process.
More discussion - see dm-devel archive.
Milan
Quoted 3 lines · Milan Broz
What is the IO scheduler for the underlying device? (You can try deadline or cfq with some tuning - this is possible to tune online).
On my system, I get the best results with these (cfq):
echo "1" > /sys/block/sda/queue/iosched/low_latency echo "8" > /sys/block/sda/queue/iosched/slice_idle echo "8" > /sys/block/sda/queue/iosched/quantum
On some systems, things improve with slice_idle set to 0.
Quoted 10 lines · Milan Broz
Quoted 3 lines · Arno Wagner
The set-up is a dm-crypt partition with a Windows XP VM and the current VMplayer, all with kernel 2.6.34.4 from kernel.org and current vmware-tools in the VM.
Can you be more specific?
You have direct partition with LUKS and this is mapped to vmware directly as disk, right?
No fs in the middle?
This is VMwarePlayer. I do not think it even supports putting the OS on a partition. So no, there is ext3 in between and the VM filesystem Image goes into several 2GB files.
Ah, forgot one possibly important detail: This is dm-crypt on top of md RAID1.
Quoted 2 lines
Are there other encrypted disks on the same underlying device?
No and there is no other didk activity at the time of these freezes.
Quoted 1 line
Underlying disk is not SSD, right?
No.
Quoted 2 lines
What is the preemption type set in Linux kernel (Processor type/Preemption model in kernel config)?
Low latency desktop. But the freezes are too long to be CPU related and there is heavy disk activity when they happen. Therefore mty conclusion that this is an emegerncy flush.
Quoted 3 lines
What is the IO scheduler for the underlying device? (You can try deadline or cfq with some tuning - this is possible to tune online).
Now you are getting into performance optimization. This is not a performance optimization issue, as there is no other load on the machine when this happens and the problem is not something being slow. Anyways, I just used the default, it has worked well for me so far.
Any recommendations for a different one?
Quoted 3 lines
Probably tuning fs inside vm can help too. If it is massive syncing with lot of data waiting for completion, it is not good.
Should not. However I do not know what stupidity can be found in the XP filesystem layer and OS. I suspect quit a lot. What I am doing when this happens is scrollign thhrough a word document that has been open for some time or making small edits with the keyboard. I really see no reasons to flush a lot of data to disk on this, but it seems it is done here.
Then there is also the VMware layer, which has its own write buffering. Turning it off or on seems to have no effect on the problem, so for the moment I assume this is not the source of the problem.
Quoted 4 lines
btw I hope that in 2.6.36 we switch to global workqueues per cpu for dm-crypt, unfortunately Alasdair is still waiting with committing it (patches are ready and I am using it myself). (It can help or it can make it worse here. It needs testing.)
I expect I can test this ;-)
Anyways, thanks and I think I will just run the OS unencrypted for the moment. The data still goes into the encrypted partition and curiously that does not cause problems. Very strange.
Arno
Signature · 7 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: [email protected] GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier
Quoted 12 lines · Arno Wagner
Quoted 10 lines · Milan Broz
Quoted 3 lines · Arno Wagner
The set-up is a dm-crypt partition with a Windows XP VM and the current VMplayer, all with kernel 2.6.34.4 from kernel.org and current vmware-tools in the VM.
Can you be more specific?
You have direct partition with LUKS and this is mapped to vmware directly as disk, right?
No fs in the middle?
Quoted 6 lines
This is VMwarePlayer. I do not think it even supports putting the OS on a partition. So no, there is ext3 in between and the VM filesystem Image goes into several 2GB files.
Ah, forgot one possibly important detail: This is dm-crypt on top of md RAID1.
This is quite complex setup...
(vmware ws suppported direct access to disk device, it is probably better setup here. Did they this functionality from player?)
I think ext3 part is problem here too, is it using barriers? (Try to switch them off, it is mount option - just to see if it helps.)
Quoted 4 lines
Low latency desktop. But the freezes are too long to be CPU related and there is heavy disk activity when they happen. Therefore mty conclusion that this is an emegerncy flush.
There can be more problems, dm-crypt itself contains cond_resched() after every encrypted sector allowing decision of scheduler to switch there. So dm-crypt istelf should not freeze host system this way, but it doesn't mean that VM suffers from starvation.
I'll check this, I am using similar system, maybe it uncovers some other problems...
Quoted 3 lines
Anyways, thanks and I think I will just run the OS unencrypted for the moment. The data still goes into the encrypted partition and curiously that does not cause problems. Very strange.
(There is also option to encrypt VM in vmware workstation7 directly. I wonder if they support it in player :-)
Milan
Quoted 13 lines · Milan Broz
Quoted 12 lines · Arno Wagner
Quoted 10 lines · Milan Broz
Quoted 3 lines · Arno Wagner
The set-up is a dm-crypt partition with a Windows XP VM and the current VMplayer, all with kernel 2.6.34.4 from kernel.org and current vmware-tools in the VM.
Can you be more specific?
You have direct partition with LUKS and this is mapped to vmware directly as disk, right?
No fs in the middle?
Quoted 8 lines
Quoted 6 lines
This is VMwarePlayer. I do not think it even supports putting the OS on a partition. So no, there is ext3 in between and the VM filesystem Image goes into several 2GB files.
Ah, forgot one possibly important detail: This is dm-crypt on top of md RAID1.
This is quite complex setup...
Yes, come to think of it, it is. It is decepticely easy to do however, since the layers all have clean interfaces.
Quoted 2 lines
(vmware ws suppported direct access to disk device, it is probably better setup here. Did they this functionality from player?)
No, I do not think so.
Quoted 2 lines
I think ext3 part is problem here too, is it using barriers? (Try to switch them off, it is mount option - just to see if it helps.)
Will try that. Thanks.
Quoted 9 lines
Quoted 4 lines
Low latency desktop. But the freezes are too long to be CPU related and there is heavy disk activity when they happen. Therefore mty conclusion that this is an emegerncy flush.
There can be more problems, dm-crypt itself contains
cond_resched()after every encrypted sector allowing decision of scheduler to switch there. So dm-crypt istelf should not freeze host system this way, but it doesn't mean that VM suffers from starvation.
Good idea. I have not noticed freezes when the focus did not belong to the VM window. Should check that as well, maybe it really is CPU starvation.
Quoted 9 lines
I'll check this, I am using similar system, maybe it uncovers some other problems...
Quoted 3 lines
Anyways, thanks and I think I will just run the OS unencrypted for the moment. The data still goes into the encrypted partition and curiously that does not cause problems. Very strange.
(There is also option to encrypt VM in vmware workstation7 directly. I wonder if they support it in player :-)
Don't think so. At least not in any obvious manner.
Arno
Signature · 7 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: [email protected] GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier
Quoted 1 line · Arno Wagner
On Wed, Aug 18, 2010 at 04:47:24PM +0200, Milan Broz wrote:
[...]
Quoted 4 lines
Quoted 2 lines
I think ext3 part is problem here too, is it using barriers? (Try to switch them off, it is mount option - just to see if it helps.)
Will try that. Thanks.
Preliminary but this seems to help a lot. Not sure the problem is entirely gone though. But if not it happens very rarely now.
What do I lose with barriers off?
Arno
Signature · 7 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: [email protected] GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier
Quoted 1 line · Arno Wagner
What do I lose with barriers off?
data security ;)
In case of the filesystem barriers (not the IO barriers, which are different AFAIK) they're used to make sure, that the COMMITS in the journal are written after the journal is correctly flushed out.
Cheers,
Chris.
Quoted 8 lines · Christoph Anton Mitterer
Quoted 1 line · Arno Wagner
What do I lose with barriers off?
data security ;)
In case of the filesystem barriers (not the IO barriers, which are different AFAIK) they're used to make sure, that the COMMITS in the journal are written after the journal is correctly flushed out.
If that means I am back to ext2 safety levels, then I can live with that.
Arno
Signature · 7 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: [email protected] GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier
Quoted 2 lines · Arno Wagner
If that means I am back to ext2 safety levels, then I can live with that.
AFAIK it's even worse (potentially).
In case you have a crash,... the kernel will (on remounting) replay the (also corrupted) journal, thereby destroying even more.
Cheers,
Chris.
So I'd suggest to rather disable the journal then...
Cheers,
Chris.
Quoted 6 lines · Arno Wagner
Quoted 3 lines
In case of the filesystem barriers (not the IO barriers, which are different AFAIK) they're used to make sure, that the COMMITS in the journal are written after the journal is correctly flushed out.
If that means I am back to ext2 safety levels, then I can live with that.
You could turn the hardware wb cache of your harddrive off and leave the barriers off too..
I've been running my disks a liftime without barrier support without any hassle, because it improves speed and interactivity a lot, but I must admit that my machines are all UPS secured.
/dev/mapper/root on / type xfs (rw,noatime,logbsize=256k,logbufs=2,nobarrier) /dev/mapper/home on /home type xfs (rw,noatime,logbsize=256k,logbufs=2,nobarrier)
Quoted 8 lines · Christoph Anton Mitterer
Quoted 1 line · Arno Wagner
What do I lose with barriers off?
data security ;)
In case of the filesystem barriers (not the IO barriers, which are different AFAIK) they're used to make sure, that the COMMITS in the journal are written after the journal is correctly flushed out.
Just disabled the journal. That should do it for the moment.
Arno
Signature · 7 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: [email protected] GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier
Quoted 8 lines · Christoph Anton Mitterer
Quoted 1 line · Arno Wagner
What do I lose with barriers off?
data security ;)
In case of the filesystem barriers (not the IO barriers, which are different AFAIK) they're used to make sure, that the COMMITS in the journal are written after the journal is correctly flushed out.
Slight confusion here.
FS uses flush (called by fsync for example), it is currently implemented using IO barrier.
After this operation, FS code can be sure that preceding barrier reached disk.
(Device-mapper internally waits for all IO to finish, processing always one barrier at a time and queuing following requests.)
If you disable it, data simply reach disk later and e.g. unexpected power loss can cause quite serious data loss.
But you probably see better performance.
So disabling barriers helps in your case? Then probably some tuning of fs can help also.
(There is ongoing discussion about reimplementing barriers in block layer, maybe it will slightly help here too.)
Milan
Quoted 6 lines · Milan Broz
Quoted 5 lines · Christoph Anton Mitterer
Quoted 1 line · Arno Wagner
What do I lose with barriers off?
data security ;)
Quoted 8 lines
Quoted 3 lines
In case of the filesystem barriers (not the IO barriers, which are different AFAIK) they're used to make sure, that the COMMITS in the journal are written after the journal is correctly flushed out.
Slight confusion here.
FS uses flush (called by fsync for example), it is currently implemented using IO barrier.
Ok. So is a process calls fsync on a file, this happens.
Quoted 4 lines
After this operation, FS code can be sure that preceding barrier reached disk.
(Device-mapper internally waits for all IO to finish, processing always one barrier at a time and queuing following requests.)
Seems there is some backlog in my case.
What I find curious is that plain ext3 on RAID1 (same disks, different partitions) does not cause problems. I would expect that an fsync blocks the disk, not only the partition. Maybe having the system and data on different filesystems just reduced the backlog enough.
Quoted 3 lines
If you disable it, data simply reach disk later and e.g. unexpected power loss can cause quite serious data loss.
But you probably see better performance.
I went back to ext2 for the time being. This is used only to work on Word documents. The data gets copied off imediately anyways and I have a backup of the complete VM in case something breaks.
Quoted 2 lines
So disabling barriers helps in your case? Then probably some tuning of fs can help also.
I would expect that. Especially more aggressive flushing should help. Will look into it when I have time. For the moment I am happy to have a solution that does not increase the considerable pain level word manages to cause all by itself (I am a LeTeX person....).
Quoted 2 lines
(There is ongoing discussion about reimplementing barriers in block layer, maybe it will slightly help here too.)
I find them quite a pain sometimes, especially when writing large amounts of data. I used to have a fine-tuned parameter set that managed to almost completely avoid emergency flushes, while having minimal impact on performance. Then the kernel devs decided to take the interface away. I am still mad at them for that. Should probably look at this again.
Arno
Signature · 7 lines
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: [email protected] GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier
Quoted 4 lines · Arno Wagner
I went back to ext2 for the time being. This is used only to work on Word documents. The data gets copied off imediately anyways and I have a backup of the complete VM in case something breaks.
You could mount ext3 with "data=journal", this gives identical behaviour.
Quoted 2 lines
Especially more aggressive flushing should help. Will look into it when I have time.
See my reply a few mails above in this thread.