Reconsidering default options for cryptsetup-reencrypt
Hi,
I'm currently reencrypting my (hardware) RAID setup, which is quite big (12 TB) in order to change the cipher from aes-xts-plain to aes-xts-plain64.
While playing around with various options, I found out that a block size of 64 MB and the use of direct I/O results in the highest speed for me.
Now, I admit that at least the "optimal" block size depends very much on your setup and the caches and/or buffers involved.
However I would argue that enabling direct I/O should be faster on most systems considering that usual block devices are probably quite big - at least compared to "normal" files and reencrypting the device is a purely consecutive process.
Running some tests I could confirm my assumption. In my case where I've got a hardware RAID with read ahead enabled, it makes at least a difference of about 10%.
Is there a reason why direct I/O is not enabled by default? Maybe I'm missing something obvious here? Personally I think that 10% can be quite much, especially when the process takes 24 hours and more.
That said I think the bigger influence is the choice of the right block size. By choosing the right block size I could double the speed. As already said above I think it probably is quite hard to get this one right automatically for each and everyone, because it depends upon various caches involved. From a theoretical stand point it probably should be about half the size of the smallest cache involved. I'm not sure whether it makes any sense (or is even possible) to probe for these things, but considering the speed enhancement of - at least in my case - 100%, there should probably be something done about it.
Now, before implementing any of this, I would like to know whether you've got similar and/or even contradicting experiences and whether there are specific reasons for the default values of the options mentioned above. Maybe I'm just generalizing too much here when trying to come up with some conclusions based on my hardware.
Best regards,
Karol Babioch
Quoted 7 lines · Karol Babioch
Hi,
I'm currently reencrypting my (hardware) RAID setup, which is quite big (12 TB) in order to change the cipher from aes-xts-plain to aes-xts-plain64.
While playing around with various options, I found out that a block size of 64 MB and the use of direct I/O results in the highest speed for me.
This seems to be RAID options (i.e. out-of-scope for cryptsetup), and very specific ones as you have some hardware RAID solution. Also 64MB block size is very, very large and will be a problem for some workloads.
Also, your sped gain of 10% is negligible, as it is likely not robust and very much dependent on the workload.
[...]
Quoted 2 lines
Maybe I'm just generalizing too much here when trying to come up with some conclusions based on my hardware.
I think you do. I also think that HW raid is a bit exotic on Linux these days, as SW RAID gives better features and often better performance.
Arno
--
Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: [email protected]
GnuPG: ID: CB5D9718 FP: 12D6 C03B 1B30 33BB 13CF B774 E35C 5FA1 CB5D 9718
----
One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision. -- Bertrand Russell
Quoted 3 lines · Arno Wagner
I think you do. I also think that HW raid is a bit exotic on Linux these days, as SW RAID gives better features and often better performance.
If you work in enterprise environment, this is definitely not true :)
And for some strange reasons, cloud environments are more and more popular where "elastic" storage means usually some virtualized HW storage array.
Also there are SSD drives which uses internally RAID0 to speed up operations (in reality it is more independent SSD drives inside).
Milan
Quoted 7 lines · Milan Broz
Quoted 3 lines · Arno Wagner
I think you do. I also think that HW raid is a bit exotic on Linux these days, as SW RAID gives better features and often better performance.
If you work in enterprise environment, this is definitely not true :)
It depends. I have used a 12-way SAS RAID controller as dumb controller with software-RAID and have gotten better results. But that was one with Linux support that sucked. With good Linux support, that may be entirely different.
On the exotic, I agree that in enterprise environments HW controllers are the norm, but I think by now that is mainly because they are simpler to use.
Quoted 3 lines
And for some strange reasons, cloud environments are more and more popular where "elastic" storage means usually some virtualized HW storage array.
Ah, yes. The "cloud" stuff, where all your data turns into hot vapor. I really do not get why people think using even more complex environments that they do not even fully control is a good idea when we obviously have not even mastered simpler systems...
Quoted 2 lines
Also there are SSD drives which uses internally RAID0 to speed up operations (in reality it is more independent SSD drives inside).
Ok, but I think _that_ does not count, unless you know of some that expose the controller. ;-)
Arno
--
Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: [email protected]
GnuPG: ID: CB5D9718 FP: 12D6 C03B 1B30 33BB 13CF B774 E35C 5FA1 CB5D 9718
----
One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision. -- Bertrand Russell
Quoted 5 lines · Arno Wagner
Quoted 2 lines
Also there are SSD drives which uses internally RAID0 to speed up operations (in reality it is more independent SSD drives inside).
Ok, but I think _that_ does not count, unless you know of some that expose the controller. ;-)
All these drives in manline linux (with patch above) are visible as separate devices (separate controllers) while with proprietary driver you see one big device (which is raid0 internally in fact)
I was able to mount it with mdadm with 64k chunk size later (with separate drives).
Its fw seems to be optimized for the raid0 case though...
Milan
Quoted 7 lines · Arno Wagner
Quoted 6 lines · Karol Babioch
Hi,
I'm currently reencrypting my (hardware) RAID setup, which is quite big (12 TB) in order to change the cipher from aes-xts-plain to aes-xts-plain64.
Quoted 8 lines
Quoted 2 lines
While playing around with various options, I found out that a block size of 64 MB and the use of direct I/O results in the highest speed for me.
This seems to be RAID options (i.e. out-of-scope for cryptsetup), and very specific ones as you have some hardware RAID solution. Also 64MB block size is very, very large and will be a problem for some workloads.
On a side note, most HW Raids do not provide IO hinting, even worse, as a result the IO hinting is wrong, which has an impact on alignment etc., esp. when it boils down to something special like 4k sector drives. If the RAID stripes are not aligned to the 4k boundaries of the drive, and dmcrypt cannot align to the 4k boundary as well as the stripe size, you will suffer a strong degradation in performance.
BTW, what exactly are you referring to, when you talk about 64 MB blocksize?
Quoted 5 lines
[...]
Arno
Quoted 13 lines
--
Arno Wagner, Dr. sc. techn., Dipl. Inform., Email: [email protected]
GnuPG: ID: CB5D9718 FP: 12D6 C03B 1B30 33BB 13CF B774 E35C 5FA1 CB5D 9718
----
One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision. -- Bertrand Russell
_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt
-Sven
Quoted 1 line · Sven Eschenberg
BTW, what exactly are you referring to, when you talk about 64 MB blocksize?
Here cryptsetup-reencrypt is in principle simple program, it reads a "block" and write it back to device with new encryption parameters (and optionally with some different offset). So block here is meant as an unit which is handled in one reencryption step.
(But the real atomic unit of encryption is still 512B block of course.)
There is no requirement this block to be aligned to underlying hw alignment (but if it is misaligned, the same performance degradation problems apply of course).
Milan
Sorry for the late reply,
just a small followup. What you are saying Milan, is absolutely true, but for misaligned HW Raid setups, a huge block size together with O_DIRECT will bypass the OS'es caching strategy and the controller (knowing the actual layout) has a way better chance of compensating unnecessary IO with it's optimized write back caching (in the linear read-modify-write case of reenrypt). For rather small block sizes (near the size of a RAID stripe) in a misaligned case, such a compensation/elimination of unneeded IO is much harder.
So small blocksize, lack of topology information, misalignment -> approx 50% loss in performance (as surveilled by opener)
Regards
-Sven
Quoted 18 lines · Milan Broz
Quoted 1 line · Sven Eschenberg
BTW, what exactly are you referring to, when you talk about 64 MB blocksize?
Here cryptsetup-reencrypt is in principle simple program, it reads a "block" and write it back to device with new encryption parameters (and optionally with some different offset). So block here is meant as an unit which is handled in one reencryption step.
(But the real atomic unit of encryption is still 512B block of course.)There is no requirement this block to be aligned to underlying hw alignment (but if it is misaligned, the same performance degradation problems apply of course).
Milan
_______________________________________________
dm-crypt mailing list
[email protected]
http://www.saout.de/mailman/listinfo/dm-crypt
Quoted 4 lines · Karol Babioch
However I would argue that enabling direct I/O should be faster on most systems considering that usual block devices are probably quite big - at least compared to "normal" files and reencrypting the device is a purely consecutive process.
But unfortunately your assumption is not correct. Not only direct hw is used, many times you are running container from file (namely for virtual machines).
I have no problem with changing defaults but I selected current defaults because of testing quite a lot of systems. But everything is configurable (and you can even restart reencryption with changed options.)
Quoted 3 lines
Running some tests I could confirm my assumption. In my case where I've got a hardware RAID with read ahead enabled, it makes at least a difference of about 10%.
Differences can be even 30-40% but both directions, depends on system. (For different block sizes even more.)
So not only direct io, also block size and sometimes enabling fsync helps too.
Quoted 9 lines
That said I think the bigger influence is the choice of the right block size. By choosing the right block size I could double the speed. As already said above I think it probably is quite hard to get this one right automatically for each and everyone, because it depends upon various caches involved. From a theoretical stand point it probably should be about half the size of the smallest cache involved. I'm not sure whether it makes any sense (or is even possible) to probe for these things, but considering the speed enhancement of - at least in my case - 100%, there should probably be something done about it.
Just few use cases I tried
- notebooks with SSD or rotational disks
- high speed PCIe SSD array
- direct rotational disk
- MD arrays of rotational disks
- external storage (with FC multipath betwen it)
- iSCSI
- VM with file backend (VMware and KVM)
...
Try it, you will see that common default doesn't work for some of these.
Anyway, defaults can be changed, we can add some heuristic or benchmark but I would like to see that tool tested more first.
Milan