Check of pool ernie/cache failed (status:1). Manual repair required!

Hello!

I have an issue re-gaining access to my data. The error messages are:

# lvchange -ay ernie/system
  Check of pool ernie/cache failed (status:1). Manual repair required!
# lvconvert --repair ernie/system
bad checksum in superblock
  Repair of cache metadata volume of cache ernie/system failed (status:1). 
Manual repair required!

How do I recover from this situation and repair the volume group?

For a full log, including the output of pvdisplay, vgdisplay and lvdisplay -a, please see attached log file. If more information is necessary, please ask.

What led to this situation:
I am using one of the infamous AMD Ryzen 2400G with an AMD B350 chipset, which suffers from random lockups related to CPU C-states [1]. With the recent AGESA 1.0.0.2a firmware update and the introduction of the "Power Supply Idle Control = Typical Current Idle" setting, the system is stable, if it boots at all. But it often takes several attempts to boot -- the failed attempts ending in weird firmware / EFI or CPU / idle related Linux kernel stack traces, which sometimes even require a hard reset, since soft-reboot (ctrl+alt +del) sometimes has no effect, because init dies. This was such a situation, where I had to reboot (ctrl+alt+del) and reset (hard) the system several times, at the end of which everything seemed fine, except that dracut was timing out when activating the disks. Debugging the situation using a Fedora 28 live system resulted in the attached log file.

--Dennis

[1]: https://bugzilla.kernel.org/show_bug.cgi?id=196683

# pvdisplay
  WARNING: Not using lvmetad because a repair command was run.
  /dev/cdrom: open failed: No medium found
  /dev/sr1: open failed: No medium found
  --- Physical volume ---
  PV Name               /dev/sdb3
  VG Name               ernie
  PV Size               <3.64 TiB / not usable 2.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              953742
  Free PE               0
  Allocated PE          953742
  PV UUID               R4qBtd-llco-Y2dj-ymFq-KxIB-lJv5-ySnjoa

--- Physical volume ---

  PV Name               /dev/nvme0n1
  VG Name               ernie
  PV Size               <232.89 GiB / not usable <3.18 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              59618
  Free PE               0
  Allocated PE          59618
  PV UUID               CFrtOF-iDtJ-FLjh-i1c7-6DEs-8K8g-K1Son5

# vgdisplay
  WARNING: Not using lvmetad because a repair command was run.
  /dev/cdrom: open failed: No medium found
  /dev/sr1: open failed: No medium found
  --- Volume group ---
  VG Name               ernie
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  131
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               1
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               <3.87 TiB
  PE Size               4.00 MiB
  Total PE              1013360
  Alloc PE / Size       1013360 / <3.87 TiB
  Free  PE / Size       0 / 0   
  VG UUID               zzTyFs-nGPU-ksKG-K37V-U2Pz-OBLs-djtFCH

# lvdisplay --all
  WARNING: Not using lvmetad because a repair command was run.
  /dev/cdrom: open failed: No medium found
  /dev/sr1: open failed: No medium found
  --- Logical volume ---
  LV Path                /dev/ernie/system
  LV Name                system
  VG Name                ernie
  LV UUID                BflUud-7t4Q-5j7W-TEg0-AJiL-gLLT-CH3l4Q
  LV Write Access        read/write
  LV Creation host, time archiso, 2018-03-06 16:13:22 -0500
  LV Cache pool name     cache
  LV Cache origin name   system_corig
  LV Status              NOT available
  LV Size                2.00 TiB
  Current LE             524288
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

--- Logical volume ---

  LV Path                /dev/ernie/data
  LV Name                data
  VG Name                ernie
  LV UUID                Ac62Oc-hkkD-tJNH-EVFl-zjd0-w5KW-VXe1vn
  LV Write Access        read/write
  LV Creation host, time ernie, 2018-03-21 21:24:00 -0400
  LV Status              available
  # open                 1
  LV Size                <1.64 TiB
  Current LE             429454
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

--- Logical volume ---

  Internal LV Name       cache
  VG Name                ernie
  LV UUID                d3w5lx-MYKd-hCmg-rOzn-22f9-QoUI-XzXjLt
  LV Write Access        read/write
  LV Creation host, time ernie, 2018-04-28 08:17:13 -0400
  LV Pool metadata       cache_cmeta
  LV Pool data           cache_cdata
  LV Status              NOT available
  LV Size                <232.79 GiB
  Current LE             59594
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

--- Logical volume ---

  Internal LV Name       lvol0_pmspare
  VG Name                ernie
  LV UUID                9XxlMS-GEjl-H56Q-Qvy8-n7vR-0jpp-la07Ka
  LV Write Access        read/write
  LV Creation host, time ernie, 2018-04-28 08:17:13 -0400
  LV Status              NOT available
  LV Size                48.00 MiB
  Current LE             12
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

--- Logical volume ---

  Internal LV Name       cache_cmeta
  VG Name                ernie
  LV UUID                oGkCUA-Do2q-zhEV-TMt7-TN8g-p25e-C0cpfG
  LV Write Access        read/write
  LV Creation host, time ernie, 2018-04-28 08:17:13 -0400
  LV Status              NOT available
  LV Size                48.00 MiB
  Current LE             12
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

--- Logical volume ---

  Internal LV Name       cache_cdata
  VG Name                ernie
  LV UUID                J6HA4c-1zbK-0yXX-gdEf-HwZi-2wDn-0GtotN
  LV Write Access        read/write
  LV Creation host, time ernie, 2018-04-28 08:17:13 -0400
  LV Status              NOT available
  LV Size                <232.79 GiB
  Current LE             59594
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

--- Logical volume ---

  Internal LV Name       system_corig
  VG Name                ernie
  LV UUID                3xrU9V-kzgH-7Ncr-AQoE-u4lv-iiki-BbZIom
  LV Write Access        read/write
  LV Creation host, time ernie, 2018-04-28 08:17:44 -0400
  LV origin of Cache LV  system
  LV Status              NOT available
  LV Size                2.00 TiB
  Current LE             524288
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

# lvchange -ay /dev/ernie/system
  WARNING: Not using lvmetad because a repair command was run.
  /dev/cdrom: open failed: No medium found
  /dev/sr1: open failed: No medium found
  Check of pool ernie/cache failed (status:1). Manual repair required!
# lvconvert --repair /dev/ernie/system
  WARNING: Disabling lvmetad cache for repair command.
  WARNING: Not using lvmetad because of repair.
  /dev/cdrom: open failed: No medium found
  /dev/sr1: open failed: No medium found
bad checksum in superblock
  Repair of cache metadata volume of cache ernie/system failed (status:1). Manual repair required!

P.S. I already found older posts [2,3] describing a similar scenario. At the time a recovery was impossible for the user, but it seems that the situation improved somewhat since then. However, I am still stick, with lvconvert -- repair asking me to repair "manually" (whatever that means), cache_dump -- repair not being able to operate on non-active LVs and LVM refusing to activate the LV as long as it has not been repaired.

[2]: https://www.redhat.com/archives/linux-lvm/2016-December/msg00013.html
[3]: https://www.redhat.com/archives/linux-lvm/2015-August/msg00008.html

Hello!

In case the question comes up: Fedora 28 (the live system I am trying to use for recovery) is using Linux 4.16.3-301.fc28 and lvm version:
LVM version 2.02.177(2)
Library version 1.02.146
Driver version 4.37.0

The system I was originally using to break the cache was using Linux 4.16.7- gentoo, and LD_LIBRARY_PATH=$PWD/lib64 ./bin/lvm version from its initrd reports (running on the Fedora 28 live system):
LVM version: 2.02.173(2)
Library version: 1.02.142
Driver version: 4.37.0

Could someone please give me a hint what is actually broken here? Is it just the metadata? If so, how can that break -- shouldn't that metadata never be modified after creating the LV? And could it not be recovered from /etc/lvm/ archive? What does "manual repair" mean in detail? Is there some way to recover the cache? Or is it at least possible to uncache the LV forcibly, to hopefully recover the data on the origin LV? What is your recommendation to minimise data loss?

Best regards,
Dennis

Dne 15.5.2018 v 10:11 Dennis Schridde napsal(a):

Hi

To get direct access to metadata - you could use your latest lvm2 2.02.177 build (or even 'git master'). In these recent versions there is added support to activate directly these 'subLVs'. So with latest lvm2 you can:

lvchange -ay vg/lv_cmeta

and then you can capture content of this LV via 'dd' into file and compress and attach 'xz' compressed to BZ.

(dd if=/dev/vg/lv_cmeta of=/tmp/file)

----

With older lvm2 you would need to 'swap-in' temporary LV as metadata, which has been also documented several in the list and man pages for thin-pool. But since you already have 2.02.177 - you should be able to activate _cmeta LV directly.

Regards

Zdenek

I tried that (using 2.02.177), but it does not work:

# lvchange -ay ernie/cache_cmeta
  Operation not permitted on hidden LV ernie/cache_cmeta.

--Dennis

Does anyone have a suggestion how I can activate this volume in order to extract the information Zdenek asked for?

For more context please see the start of this thread. The gist is that my cached LV cannot be activated anymore, and lvconvert --repair reports:
bad checksum in superblock
Repair of cache metadata volume of cache ernie/system failed (status:1).
Manual repair required!

My most important questions are:
* What is broken?
- What information does the superblock carry / what is its purpose?
- Where is it located / which part of my disk was damaged?
- What will be the consequence of it being irrecoverably lost?
* What does "manual repair" mean in detail?
- Using a specific tool?
- Flipping bits using a hex editor?
* Is there some way to recover the cache? Or is it at least possible to uncache the LV forcibly, to hopefully recover the data on the origin LV?
* What is your recommendation to minimise data loss?

--Dennis

How would I create a backup of the affected LVs, so that dangerous commands would not destroy the data and get me into an unrecoverable state? Is that possible without activating the volumes? Is it possible to force-activate the volumes, without changing the bits on the disk -- just so that I can read from the LV and create the backup?

--Dennis

Hello!

This command was only successful in LVM 2.02.178-rc1, but failed in 2.02.177.

I created a report in bugzilla [4], but since the cache_cmeta LV appears to contain parts of my personal files, I cannot attach it. However, I gained access to the metadata LV now and would be able to provide you access to certain specific parts of it, if requested.

My goal is still to gain access to as much of the data as possible.

--Dennis

[4]: https://bugzilla.redhat.com/show_bug.cgi?id=1585670

It definitely should not. _cdata is where fragments you your data are stored, and _cmeta contains only metadata (e.g. counters and references to cdata.)

If there really are fragments of data in _cmeta LV, something must have gone wrong elsewhere.

Thanks for the insight. I attached the first 4890783 bytes (i.e. those before the blocks of seemingly unstructured data, which I saw in hexdump -C) of that LV to the bug report [4]. Hopefully it contains enough information to figure out what is broken.

[4]: https://bugzilla.redhat.com/show_bug.cgi?id=1585670