Lvmetad crashes with "memory leak" during parallel lvm operations

Question

Lvmetad crashes with "memory leak" during parallel lvm operations

I have a problem with LVMETAD crashing during parallel lvm operations (vgscan, pvscan / lvscan / lvs / vgs)

After applying debugging on lvmetad, I could see that the operation vg_list

overlapped with another thread and caused a linked list leak and subsequent "memory splitting" for LVMETAD.

>Jul 17 15:04:14 NODE1 lvmeta.sh[4623]: token_update end len 3 pid 5763 new token filter:3239235440
>Jul 17 15:04:14 NODE1 lvmeta.sh[4623]: 5763 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 17 15:04:14 NODE1 lvmeta.sh[4623]: 5763 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:06:46 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:06:46 NODE1 lvmeta.sh[4623]: pv_list
>Jul 18 12:06:47 NODE1 lvmeta.sh[4623]: vg_lookup vgid wxfglR-s3f8-yc3z-sg8K-JcBi-o0Qy-Ic8z0w name VG_NODE1
>Jul 18 12:06:47 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid wxfglR-s3f8-yc3z-sg8K-JcBi-o0Qy-Ic8z0w
>Jul 18 12:06:48 NODE1 lvmeta.sh[4623]:   PV /dev/sda4   VG VG_NODE1       lvm2 [<106.71 GiB / 47.75 GiB free]
>Jul 18 12:06:48 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:06:48 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:06:48 NODE1 lvmeta.sh[4623]:   PV /dev/sda3   VG VG_System       lvm2 [29.29 GiB / 9.60 GiB free]
>Jul 18 12:06:48 NODE1 lvmeta.sh[4623]:   Total: 2 [136.00 GiB] / in use: 2 [136.00 GiB] / in no VG: 0 [0   ]
>Jul 18 12:07:01 NODE1 lvmeta.sh[4623]: 9936 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:01 NODE1 lvmeta.sh[4623]: 9936 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:01 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 9977 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 9977 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10003 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10003 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10030 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10030 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10055 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10055 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10083 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10083 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10102 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10102 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10129 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10129 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10153 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: 10153 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:02 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10177 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10177 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10200 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10200 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10225 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10225 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10250 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10250 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10276 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10276 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10297 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10297 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10324 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10324 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10344 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10344 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10360 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10360 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10375 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: 10375 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:03 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10395 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10395 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10424 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10424 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10426 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10426 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10438 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10438 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10446 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10446 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10456 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10456 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10467 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10467 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10484 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: 10484 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:04 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10502 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10502 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10536 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10536 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10551 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10551 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_clear_outdated_pvs vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: set info VG name none uuid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 cache_version 351 new_version 352
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10574 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10574 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: set info VG name none uuid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 cache_version 351 new_version 353
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10590 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10590 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_list
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_update vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: update_metadata begin arg_vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 arg_name VG_System pvid none
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: update_metadata existing vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 and existing name VG_System
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: update_metadata for VG_System ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 from 351 to 353
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: moving PV tpUO22-Ra0R-sXa4-bFH4-eFH1-z2Dr-Nf46dV to VG #orphan
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: moving PV tpUO22-Ra0R-sXa4-bFH4-eFH1-z2Dr-Nf46dV to VG ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: update_metadata is done for VG_System ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: pv_found pvid tpUO22-Ra0R-sXa4-bFH4-eFH1-z2Dr-Nf46dV on device 2051 matches existing
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: vg_lookup vgid ziVnMV-iS3r-PMBW-VVKO-pE9F-Q49h-5LWne2 name VG_System
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: 10649 global info flags none reason none token filter:3239235440 update_pid 0
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: *** Error in `/usr/bin/lvmetad': malloc(): smallbin double linked list corrupted: 0x000000ffdc0008c0 ***
>Jul 18 12:07:05 NODE1 lvmeta.sh[4623]: lvmetad: mm/dbg_malloc.c:271: dm_bounds_check_debug: Assertion `!"Memory smash"' failed.
>Jul 18 12:07:06 NODE1 lvmeta.sh[4623]: /usr/bin/lvmeta.sh: line 5:  5761 Aborted                 (core dumped) /usr/bin/lvmetad -f -s /run/lvm/lvmetad.socket -l debug

For me this is reproducible every time we restart my host. During, but it crashes as the systemd startup script asks for parallel vgs / lvs to be loaded into software.

Reproducing this is very simple: