Skip to contentSkip to conversation

LUKS2 SSD swap and system freeze

dm-crypt Germano Massullo 3 participants 3 messages Dec 15, 2020 – Dec 22, 2020

lore
Share & tools
lore
#0 lore raw

Good day. I am investigating about a weird behavior of my system that leads to frequent freezes.
The main components of my system are: Fedora 33 (kernel 5.9.13-200.fc33.x86_64), cryptsetup 2.3.4, an AMD Ryzen 5 3600, 16 GB RAM 3200 MHz, a Samsung 860 PRO used for 32 GB swap and LVM Cache, that is encrypted with following criteria:

# cryptsetup status ssd_cache
/dev/mapper/ssd_cache is active and is in use.
  type:    LUKS2
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: keyring
  device:  /dev/sdc
  sector size:  512
  offset:  32768 sectors
  size:    500085424 sectors
  mode:    read/write
  flags:   discards

During write bursts of swap (for example when booting some virtual machines), the system completely freezes, I cannot even move the mouse pointer. The system may remain in stuck state even for 10 minutes. When the (host) system turns back to normal, I can see for example in virtual machines, the following error in console
========
Message from syslogd@localhost at Dec 15 01:14:32 ... kernel:watchdog: BUG: soft lockup - CPU#0 stuck for 124s! [0x564a0d4baad0-:1924]
========

To check if the complete stuck of the system was caused by encryption, I tried to disable SSD swap, and create a unencrypted swap on a regular HDD, and then when I had write bursts, the system slowed, but it did not freeze.
How could I proceed in my investigation to find out if it is a matter of system configuration, a bug, etc.?
Thank you

#1 lore raw

On 15 Dec 2020 11:42 +0100, from [email protected] (Germano Massullo):

Quoted 4 lines

To check if the complete stuck of the system was caused by encryption, I tried to disable SSD swap, and create a unencrypted swap on a regular HDD, and then when I had write bursts, the system slowed, but it did not freeze.

Note that by doing so, you actually changed at least two things at
once: absence/presence of encryption, and SSD/HDD. There is also a possibility that you changed the properties of the storage interconnect. You've shown that by changing all at once, you can eliminate or introduce the problem, but you haven't unequivocally shown which specific change made the difference.

To eliminate those factors and further narrow down the problem, can you change just _one_ of those at a time? For example, do you still see the same issue with similarly encrypted swap on the HDD, or with unencrypted swap on the SSD? The answer would help at least somewhat to lay the blame on either the storage device or the cryptography.

Also, have you checked the SMART data for the SSD?

Another thing that could be worth testing is to access that system over the network. Assuming your firewall settings allow ICMP, does it respond to pings while appearing frozen, for example?

Signature · 2 lines
Michael Kjörling • https://michael.kjorling.se • [email protected]
 “Remember when, on the Internet, nobody cared that you were a dog?”
#2 lore raw
In reply to Germano Massullo
Quoted 31 lines · Germano Massullo

Good day. I am investigating about a weird behavior of my system that leads to frequent freezes.
The main components of my system are: Fedora 33 (kernel 5.9.13-200.fc33.x86_64), cryptsetup 2.3.4, an AMD Ryzen 5 3600, 16 GB RAM 3200 MHz, a Samsung 860 PRO used for 32 GB swap and LVM Cache, that is encrypted with following criteria:

# cryptsetup status ssd_cache
/dev/mapper/ssd_cache is active and is in use.
  type:    LUKS2
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: keyring
  device:  /dev/sdc
  sector size:  512
  offset:  32768 sectors
  size:    500085424 sectors
  mode:    read/write
  flags:   discards

During write bursts of swap (for example when booting some virtual machines), the system completely freezes, I cannot even move the mouse pointer. The system may remain in stuck state even for 10 minutes. When the (host) system turns back to normal, I can see for example in virtual machines, the following error in console
========
Message from syslogd@localhost at Dec 15 01:14:32 ... kernel:watchdog: BUG: soft lockup - CPU#0 stuck for 124s! [0x564a0d4baad0-:1924]
========

Quoted 6 lines

To check if the complete stuck of the system was caused by encryption, I tried to disable SSD swap, and create a unencrypted swap on a regular HDD, and then when I had write bursts, the system slowed, but it did not freeze.
How could I proceed in my investigation to find out if it is a matter of system configuration, a bug, etc.?

First, this is kernel issue, not a cryptsetup userspace issue (cryptsetup only configures kernel dm-crypt).

With some encryption drivers that allocates memory during encryption there was a possible deadlock if swap is encrypted and system is out of memory.

There should be a workaround in latest stable kernel that does not allow to use such crypto drivers in dm-crypt.
For dm-crypt it is commit
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/md/dm-crypt.c?id=cd74693870fb748d812867ba49af733d689a3604
(but it requires also patches for crypto subsystem for setting this flag in drivers.)

The long-term solution is to fix crypto drivers...

Anyway, please try the latest stable kernel (ideally 5.10.x) - if you still see the problem, please report it to [email protected] mailing list.

(There is another patch that limits IOs in flight in dm-crypt, but this was not yet merged upstream - that could help too).

Thanks,
Milan
_______________________________________________
dm-crypt mailing list
[email protected]
https://www.saout.de/mailman/listinfo/dm-crypt