Skip to contentSkip to conversation

lvmetad crashes with “memory smash” during lvm parallel operations

linux-lvm Shivarama Krishnan, Gururaj (Nokia - IN/Bangalore) 2 participants 3 messages Jul 19, 2017 – Jul 24, 2017

lore
Share & tools
lore

Shivarama Krishnan, Gururaj (Nokia - IN/Bangalore)

author
#0 lore raw

Hi ,

Greetings.

We are facing an issue with LVMETAD crashing during the parallel lvm operations (vgscan/pvscan/lvscan/lvs/vgs)

PKG_NAME=LVM2
PKG_VERSION=2.02.168
PKG_URL=LVM2.2.02.168.tgz
PKG_MD5=00209bdd6befe9a7330f162909313ae8
PKG_DATE=2016-12-01

This issue is also seen in LVM2-2.02.171.

The issue is reproduced on the MIPS Architecture HARDWARE with SCSI DISKS.

Disk information:
Device: TOSHIBA  AL13SEB600       Version: 4701
Serial number: 54R0A04HFYP6
Device type: disk
Transport protocol: SAS

Device driver :

# modinfo mptsas
filename:       /lib/modules/2.6.34.15-WR4.3.fp_octwnd_6880_standard-00701-
g75d10fa/kernel/drivers/message/fusion/mptsas.ko
version:        3.04.14
license:        GPL
description:    Fusion MPT SAS Host driver
author:         LSI Corporation
srcversion:     AF0ADD21F2C0510BB52D860
alias:          pci:v00001000d00000062sv*sd*bc*sc*i*
alias:          pci:v00001000d00000058sv*sd*bc*sc*i*
alias:          pci:v00001000d00000056sv*sd*bc*sc*i*
alias:          pci:v00001000d00000054sv*sd*bc*sc*i*
alias:          pci:v00001000d00000050sv*sd*bc*sc*i*
depends:        mptscsih,mptbase
vermagic:       2.6.34.15-WR4.3.fp_octwnd_6880_standard-00701-g75d10fa SMP preempt mod_unload OCTEON
64BIT 
parm:           mpt_pt_clear: Clear persistency table: enable=1  (default=MPTSCSIH_PT_CLEAR=0) (int)
parm:           max_lun: max lun, default=16895  (int)

So we enabled the debug flag to see the   lvmetad operation . I could see that the vg_list operation is overlapping the from the other thread and causing the leak in linked list and consequent "memory smash" for LVMETAD.

Quoted 2 lines
Jul 17 15:04:14 NODE1 lvmeta.sh[4623]: token_update end len 3 pid 5763 new token filter:3239235440
Jul 17 15:04:14 NODE1 lvmeta.sh[4623]: 5763 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 17 15:04:14 NODE1 lvmeta.sh[4623]: 5763 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:06:46 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:06:46 NODE1 lvmeta.sh[4623]: pv_list
Jul 18 12:06:47 NODE1 lvmeta.sh[4623]: vg_lookup vgid wxfglR-s3f8-yc3z-sg8K-JcBi-o0Qy-Ic8z0w name

VG_NODE1

Quoted 1 line
Jul 18 12:06:47 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid wxfglR-s3f8-yc3z-sg8K-JcBi-o0Qy-

Ic8z0w

Quoted 1 line
Jul 18 12:06:48 NODE1 lvmeta.sh[4623]:   PV /dev/sda4   VG VG_NODE1       lvm2 [<106.71 GiB / 47.75

GiB free]

Quoted 1 line
Jul 18 12:06:48 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 1 line
Jul 18 12:06:48 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2

Quoted 1 line
Jul 18 12:06:48 NODE1 lvmeta.sh[4623]:   PV /dev/sda3   VG VG_System       lvm2 [29.29 GiB / 9.60

GiB free]

Quoted 1 line
Jul 18 12:06:48 NODE1 lvmeta.sh[4623]:   Total: 2 [136.00 GiB] / in use: 2 [136.00 GiB] / in no VG:

0 [0   ]

Quoted 1 line
Jul 18 12:07:01 NODE1 lvmeta.sh[4623]: 9936 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:01 NODE1 lvmeta.sh[4623]: 9936 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:01 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 9977 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 9977 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10003 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10003 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10030 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10030 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10055 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10055 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10083 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10083 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10102 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10102 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10129 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10129 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10153 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10153 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10177 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10177 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10200 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10200 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10225 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10225 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10250 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10250 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10276 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10276 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10297 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10297 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10324 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10324 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10344 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10344 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10360 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10360 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10375 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10375 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10395 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10395 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10424 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10424 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10426 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10426 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10438 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10438 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10446 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10446 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10456 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10456 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10467 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10467 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10484 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10484 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10502 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10502 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10536 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10536 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10551 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10551 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: set info VG name none uuid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2 cache_version 351 new_version 352

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10574 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10574 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: set info VG name none uuid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-

5LWne2 cache_version 351 new_version 353

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10590 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10590 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_update vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: update_metadata begin arg_vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-5LWne2 arg_name VG_System pvid none

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: update_metadata existing vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-5LWne2 and existing name VG_System

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: update_metadata for VG_System ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-5LWne2 from 351 to 353

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: moving PV tpUO22-Ra0R-sXa4-bFH4-eFH1-z2Dr-Nf46dV to VG

#orphan

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: moving PV tpUO22-Ra0R-sXa4-bFH4-eFH1-z2Dr-Nf46dV to VG

ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: update_metadata is done for VG_System ziVnMV-iS3r-PMBW-VVKO-

pE9F-Q49h-5LWne2

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: pv_found pvid tpUO22-Ra0R-sXa4-bFH4-eFH1-z2Dr-Nf46dV on

device 2051 matches existing

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name

VG_System

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10649 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: *** Error in `/usr/bin/lvmetad': malloc(): 	 double

linked list corrupted: 0x000000ffdc0008c0 ***

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: lvmetad: mm/dbg_malloc.c:271: dm_bounds_check_debug:

Assertion `!"Memory smash"' failed.

Quoted 1 line
Jul 18 12:07:06 NODE1 lvmeta.sh[4623]: /usr/bin/lvmeta.sh: line 5:  5761

Aborted                 (core dumped) /usr/bin/lvmetad -f -s /run/lvm/lvmetad.socket -l debug

For us it is reproducible every time when we restart the my host. During but it crashes, since the systemd startup script request parallel vgs/lvs to load to the software.

Reproduction of this is very simple:

for i in seq 1 100; do lvs & done
This results in crash of lvmetad. on the mentioned version.

Attaching the gdb information (We have full gdb information available in the attachment)

Quoted 4 lines

Thread 1 (LWP 6835):
#0  __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:58 #1  0x000000fff3f53b2c in __GI_abort () at abort.c:89
#2  0x000000fff3f4910c in __assert_fail_base (fmt=0xfff4081c60 "%s%s%s:%u: %s%sAssertion `%s'

failed.\n%n",

Quoted 1 line

    assertion=0xfff4179be8 "!\"Memory smash\"", file=0xfff4179a70 "mm/dbg_malloc.c",

line=<optimized out>, function=<optimized out>) at assert.c:92

Quoted 1 line

#3  0x000000fff3f491dc in __GI___assert_fail (assertion=0xfff4179be8 "!\"Memory smash\"",

file=0xfff4179a70 "mm/dbg_malloc.c", line=<optimized out>,

Quoted 11 lines

    function=0xfff4179c08 "dm_bounds_check_debug") at assert.c:101
#4  0x000000fff415eab4 in dm_bounds_check_debug () from /usr/lib64/libdevmapper.so.1.02 #5  0x000000fff415ef04 in dm_bounds_check_wrapper () from /usr/lib64/libdevmapper.so.1.02 #6  0x000000fff415e0f0 in dm_malloc_aux_debug () from /usr/lib64/libdevmapper.so.1.02 #7  0x000000fff415ecb4 in dm_malloc_wrapper () from /usr/lib64/libdevmapper.so.1.02 #8  0x000000fff4159d2c in dm_vasprintf () from /usr/lib64/libdevmapper.so.1.02 #9  0x000000fff4159f68 in dm_asprintf () from /usr/lib64/libdevmapper.so.1.02 #10 0x000000aab47155fc in buffer_append_vf ()
#11 0x000000aab4712a80 in daemon_reply_simple ()
#12 0x000000aab470ec88 in _ftext ()
Backtrace stopped: frame did not save the PC

Can you help us on this .

With Regards,
Gururaj S
"We share We grow"

Signature · 5 lines
With Regards,
Gururaj S
FlexiPlatform Technical Service
NOKIA NETWORKS
"We share We Grow"
Quoted 83 lines

[Current thread is 1 (LWP 6835)]
(gdb)
(gdb) t a a bt

Thread 6 (LWP 6837):
#0 0x000000fff40d0648 in read () at ../sysdeps/unix/syscall-template.S:84 #1 0x000000aab4714f34 in buffer_read ()
warning: GDB can't find the start of the function at 0xaab4712cb3.

GDB is unable to find the start of the function at 0xaab4712cb3
and thus can't determine the size of that function's stack frame. This means that GDB may be unable to access that stack frame, or the frames below it.
This problem is most likely caused by an invalid program counter or
stack pointer.
However, if you think GDB should simply search farther back
from 0xaab4712cb3 for code which looks like the beginning of a function, you can increase the range of the search using the `set heuristic-fence-post' command.
#2 0x000000aab4712cb4 in ?? ()

Thread 5 (LWP 6838):
#0 0x000000fff40102d8 in mmap64 () at ../sysdeps/unix/syscall-template.S:84
#1 0x000000fff3f490a4 in __assert_fail_base (fmt=0xfff4081c60 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=0xfff4179be8 "!\"Memory smash\"", file=0xfff4179a70 "mm/dbg_malloc.c", line=<optimized out>, function=<optimized out>) at assert.c:68
#2 0x000000fff3f491dc in __GI___assert_fail (assertion=0xfff4179be8 "!\"Memory smash\"", file=0xfff4179a70 "mm/dbg_malloc.c", line=<optimized out>,
function=0xfff4179c08 "dm_bounds_check_debug") at assert.c:101

#3  0x000000fff415eab4 in dm_bounds_check_debug () from /usr/lib64/libdevmapper.so.1.02
#4  0x000000fff415ef04 in dm_bounds_check_wrapper () from /usr/lib64/libdevmapper.so.1.02
#5  0x000000fff415e424 in dm_free_aux () from /usr/lib64/libdevmapper.so.1.02
#6  0x000000fff415ede8 in dm_free_wrapper () from /usr/lib64/libdevmapper.so.1.02
#7  0x000000aab471578c in buffer_append_vf ()
#8  0x000000aab4712a80 in daemon_reply_simple ()

warning: GDB can't find the start of the function at 0xaab4712bdb. #9 0x000000aab4712bdc in ?? ()

Thread 4 (LWP 6839):

#0  0x000000fff3f51fe4 in __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:53
#1  0x000000fff3f53b2c in __GI_abort () at abort.c:89
#2  0x000000fff3f4910c in __assert_fail_base (fmt=0xfff4081c60 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",

assertion=0xfff4179be8 "!\"Memory smash\"", file=0xfff4179a70 "mm/dbg_malloc.c", line=<optimized out>, function=<optimized out>) at assert.c:92
#3 0x000000fff3f491dc in __GI___assert_fail (assertion=0xfff4179be8 "!\"Memory smash\"", file=0xfff4179a70 "mm/dbg_malloc.c", line=<optimized out>,
function=0xfff4179c08 "dm_bounds_check_debug") at assert.c:101
#4 0x000000fff415eab4 in dm_bounds_check_debug () from /usr/lib64/libdevmapper.so.1.02
#5 0x000000fff415ef04 in dm_bounds_check_wrapper () from /usr/lib64/libdevmapper.so.1.02
---Type <return> to continue, or q <return> to quit---

#6  0x000000fff415e0f0 in dm_malloc_aux_debug () from /usr/lib64/libdevmapper.so.1.02
#7  0x000000fff415e744 in dm_realloc_aux () from /usr/lib64/libdevmapper.so.1.02
#8  0x000000fff415ee6c in dm_realloc_wrapper () from /usr/lib64/libdevmapper.so.1.02
#9  0x000000aab47169f4 in buffer_realloc ()

#10 0x000000aab4714ed8 in buffer_read ()
warning: GDB can't find the start of the function at 0xaab4712cb3. #11 0x000000aab4712cb4 in ?? ()

Thread 3 (LWP 6237):

#0  0x000000fff400bd78 in select () at ../sysdeps/unix/syscall-template.S:84
#1  0x000000aab4713c70 in daemon_start ()
#2  0x000000aab4711514 in main ()

Thread 2 (LWP 6834):
#0 0x000000fff40d0648 in read () at ../sysdeps/unix/syscall-template.S:84 #1 0x000000aab4714f34 in buffer_read ()
warning: GDB can't find the start of the function at 0xaab4712cb3. #2 0x000000aab4712cb4 in ?? ()

Thread 1 (LWP 6835):

#0  __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:58
#1  0x000000fff3f53b2c in __GI_abort () at abort.c:89
#2  0x000000fff3f4910c in __assert_fail_base (fmt=0xfff4081c60 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
   assertion=0xfff4179be8 "!\"Memory smash\"", file=0xfff4179a70 "mm/dbg_malloc.c", line=<optimized out>, function=<optimized out>) at assert.c:92
#3  0x000000fff3f491dc in __GI___assert_fail (assertion=0xfff4179be8 "!\"Memory smash\"", file=0xfff4179a70 "mm/dbg_malloc.c", line=<optimized out>, 
   function=0xfff4179c08 "dm_bounds_check_debug") at assert.c:101
#4  0x000000fff415eab4 in dm_bounds_check_debug () from /usr/lib64/libdevmapper.so.1.02
#5  0x000000fff415ef04 in dm_bounds_check_wrapper () from /usr/lib64/libdevmapper.so.1.02
#6  0x000000fff415e0f0 in dm_malloc_aux_debug () from /usr/lib64/libdevmapper.so.1.02
#7  0x000000fff415ecb4 in dm_malloc_wrapper () from /usr/lib64/libdevmapper.so.1.02
#8  0x000000fff4159d2c in dm_vasprintf () from /usr/lib64/libdevmapper.so.1.02
#9  0x000000fff4159f68 in dm_asprintf () from /usr/lib64/libdevmapper.so.1.02

#10 0x000000aab47155fc in buffer_append_vf ()
#11 0x000000aab4712a80 in daemon_reply_simple ()
#12 0x000000aab470ec88 in _ftext ()
Backtrace stopped: frame did not save the PC

author
#1 lore raw

Hi,

We went through the code on the this issue and able to point out that the structure memblock in unprotected from parallel threads.

void dm_bounds_check_debug(void)

{
        struct memblock *mb = _head;
        while (mb) {
                size_t i;
                char *ptr = ((char *) (mb + 1)) + mb->length;
                for (i = 0; i < sizeof(unsigned long); i++)
                        if (*ptr++ != (char) mb->id)
                                assert(!"Memory smash");

                mb = mb->next;
        }
}

So, if in case another thread try to access remove an entry and at the same time of query of the list, the lvmetad will crash. It is good have a lock introduced to memoryblock structure.

With Regards,
Gururaj S
"We share We Grow"

Quoted 525 lines · Gururaj

Hi ,

Greetings.

We are facing an issue with LVMETAD crashing during the parallel lvm operations (vgscan/pvscan/lvscan/lvs/vgs)

PKG_NAME=LVM2
PKG_VERSION=2.02.168
PKG_URL=LVM2.2.02.168.tgz
PKG_MD5=00209bdd6befe9a7330f162909313ae8
PKG_DATE=2016-12-01

This issue is also seen in LVM2-2.02.171.

The issue is reproduced on the MIPS Architecture HARDWARE with SCSI DISKS.

Disk information:
Device: TOSHIBA AL13SEB600 Version: 4701
Serial number: 54R0A04HFYP6
Device type: disk
Transport protocol: SAS

Device driver :

# modinfo mptsas
filename:       /lib/modules/2.6.34.15
g75d10fa/kernel/drivers/message/fusion/mptsas.ko
version:        3.04.14
license:        GPL
description:    Fusion MPT SAS Host driver
author:         LSI Corporation
srcversion:     AF0ADD21F2C0510BB52D860
alias:          pci:v00001000d00000062sv*sd*bc*sc*i*
alias:          pci:v00001000d00000058sv*sd*bc*sc*i*
alias:          pci:v00001000d00000056sv*sd*bc*sc*i*
alias:          pci:v00001000d00000054sv*sd*bc*sc*i*
alias:          pci:v00001000d00000050sv*sd*bc*sc*i*
depends:        mptscsih,mptbase
vermagic:       2.6.34.15
64BIT 
parm:           mpt_pt_clear: Clear persistency table: enable=1  (default=MPTSCSIH_PT_CLEAR=0)
(int)
parm:           max_lun: max lun, default=16895  (int)

So we enabled the debug flag to see the lvmetad operation . I could see that the vg_list
operation
is overlapping the from the other thread and causing the leak in linked list and consequent "memory
smash" for LVMETAD.

Quoted 3 lines
Jul 17 15:04:14 NODE1 lvmeta.sh[4623]: token_update end len 3 pid 5763 new token
filter:3239235440
Jul 17 15:04:14 NODE1 lvmeta.sh[4623]: 5763 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 17 15:04:14 NODE1 lvmeta.sh[4623]: 5763 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 4 lines
Jul 18 12:06:46 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:06:46 NODE1 lvmeta.sh[4623]: pv_list
Jul 18 12:06:47 NODE1 lvmeta.sh[4623]: vg_lookup vgid wxfglR-s3f8-yc3z-sg8K-JcBi-o0Qy-Ic8z0w

name

VG_NODE1

Quoted 2 lines
Jul 18 12:06:47 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid wxfglR-s3f8-yc3z-sg8K-JcBi-

o0Qy-

Ic8z0w

Quoted 2 lines
Jul 18 12:06:48 NODE1 lvmeta.sh[4623]:   PV /dev/sda4   VG VG_NODE1       lvm2 [<106.71 GiB /

47.75

GiB free]

Quoted 2 lines
Jul 18 12:06:48 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 2 lines
Jul 18 12:06:48 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2

Quoted 2 lines
Jul 18 12:06:48 NODE1 lvmeta.sh[4623]:   PV /dev/sda3   VG VG_System       lvm2 [29.29 GiB /

9.60

GiB free]

Quoted 2 lines
Jul 18 12:06:48 NODE1 lvmeta.sh[4623]:   Total: 2 [136.00 GiB] / in use: 2 [136.00 GiB] / in no

VG:

0 [0 ]

Quoted 1 line
Jul 18 12:07:01 NODE1 lvmeta.sh[4623]: 9936 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:01 NODE1 lvmeta.sh[4623]: 9936 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:01 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 9977 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 9977 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10003 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10003 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10030 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10030 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10055 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10055 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10083 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10083 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10102 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10102 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10129 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10129 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10153 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10153 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10177 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10177 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 2 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10200 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10200 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10225 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10225 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10250 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10250 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10276 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10276 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10297 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10297 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10324 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10324 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2

Quoted 2 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10344 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10344 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10360 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10360 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10375 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10375 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10395 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10395 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10424 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10424 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10426 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10426 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 3 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10438 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10438 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10446 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10446 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10456 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10456 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10467 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10467 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2

Quoted 2 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10484 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10484 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10502 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10502 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10536 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10536 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10551 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10551 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2

Quoted 2 lines
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: set info VG name none uuid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2 cache_version 351 new_version 352

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10574 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10574 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: set info VG name none uuid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-

5LWne2 cache_version 351 new_version 353

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10590 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10590 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 3 lines
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_update vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 2 lines
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: update_metadata begin arg_vgid ziVnMV-iS3r-PMBW-VVKO-

pE9F-

Q49h-5LWne2 arg_name VG_System pvid none

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: update_metadata existing vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-5LWne2 and existing name VG_System

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: update_metadata for VG_System ziVnMV-iS3r-PMBW-VVKO-pE9F-

Q49h-5LWne2 from 351 to 353

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: moving PV tpUO22-Ra0R-sXa4-bFH4-eFH1-z2Dr-Nf46dV to VG

#orphan

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: moving PV tpUO22-Ra0R-sXa4-bFH4-eFH1-z2Dr-Nf46dV to VG

ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

Quoted 2 lines
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: update_metadata is done for VG_System ziVnMV-iS3r-PMBW-

VVKO-

pE9F-Q49h-5LWne2

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: pv_found pvid tpUO22-Ra0R-sXa4-bFH4-eFH1-z2Dr-Nf46dV on

device 2051 matches existing

Quoted 2 lines
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2

name

VG_System

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10649 global info flags none reason none token

filter:3239235440 update_pid 0

Quoted 2 lines
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: *** Error in `/usr/bin/lvmetad': malloc(): 	 doubl

e

linked list corrupted: 0x000000ffdc0008c0 ***

Quoted 1 line
Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: lvmetad: mm/dbg_malloc.c:271: dm_bounds_check_debug:

Assertion `!"Memory smash"' failed.

Quoted 1 line
Jul 18 12:07:06 NODE1 lvmeta.sh[4623]: /usr/bin/lvmeta.sh: line 5:  5761

Aborted (core dumped) /usr/bin/lvmetad -f -s /run/lvm/lvmetad.socket -l debug

For us it is reproducible every time when we restart the my host. During but it crashes, since the systemd startup script request parallel vgs/lvs to load to the software.

Reproduction of this is very simple:

for i in seq 1 100; do lvs & done
This results in crash of lvmetad. on the mentioned version.

Attaching the gdb information (We have full gdb information available in the attachment)

Quoted 4 lines

Thread 1 (LWP 6835):

#0  __GI_raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:58
#1  0x000000fff3f53b2c in __GI_abort () at abort.c:89
#2  0x000000fff3f4910c in __assert_fail_base (fmt=0xfff4081c60 "%s%s%s:%u: %s%sAssertion `%s'

failed.\n%n",

Quoted 1 line
    assertion=0xfff4179be8 "!\"Memory smash\"", file=0xfff4179a70 "mm/dbg_malloc.c",

line=<optimized out>, function=<optimized out>) at assert.c:92

Quoted 1 line

#3 0x000000fff3f491dc in __GI___assert_fail (assertion=0xfff4179be8 "!\"Memory smash\"",

file=0xfff4179a70 "mm/dbg_malloc.c", line=<optimized out>,

Quoted 11 lines
    function=0xfff4179c08 "dm_bounds_check_debug") at assert.c:101
#4  0x000000fff415eab4 in dm_bounds_check_debug () from /usr/lib64/libdevmapper.so.1.02
#5  0x000000fff415ef04 in dm_bounds_check_wrapper () from /usr/lib64/libdevmapper.so.1.02
#6  0x000000fff415e0f0 in dm_malloc_aux_debug () from /usr/lib64/libdevmapper.so.1.02
#7  0x000000fff415ecb4 in dm_malloc_wrapper () from /usr/lib64/libdevmapper.so.1.02
#8  0x000000fff4159d2c in dm_vasprintf () from /usr/lib64/libdevmapper.so.1.02
#9  0x000000fff4159f68 in dm_asprintf () from /usr/lib64/libdevmapper.so.1.02

#10 0x000000aab47155fc in buffer_append_vf () #11 0x000000aab4712a80 in daemon_reply_simple () #12 0x000000aab470ec88 in _ftext () Backtrace stopped: frame did not save the PC

Can you help us on this .

With Regards,
Gururaj S
"We share We grow"

#2 lore raw
In reply to Shivarama Krishnan, Gururaj (Nokia - IN/Bangalore)
Quoted 7 lines · Gururaj (Nokia - IN/Bangalore)

Hi ,

Greetings.

We are facing an issue with LVMETAD crashing during the parallel lvm operations (vgscan/pvscan/lvscan/lvs/vgs)

Quoted 7 lines

PKG_NAME=LVM2
PKG_VERSION=2.02.168
PKG_URL=LVM2.2.02.168.tgz
PKG_MD5=00209bdd6befe9a7330f162909313ae8
PKG_DATE=2016-12-01

This issue is also seen in LVM2-2.02.171.

Please check if this fix is applied to your version:
https://sourceware.org/git/?p=lvm2.git;a=commit;h=c459f23565e86eb9a8a116995cda7dcaed858b9f

AFAICT all the problems you sent are the fault of the memory debugging code, not the original problem. You need to compile lvmetad with debugging, but not memory debugging. Then gdb should give relevant backtraces.

Dave