Skip to contentSkip to conversation

potential locking issues

linux-lvm Jaco Kroon 1 participant 2 messages Feb 20, 2013 – Mar 2, 2013

lore
Share & tools
lore

Jaco Kroon

author
#0 lore raw

Hi All,

LVM2 uses a locking scheme, relying on flock to maintain lock files for volume groups, by default /var/lock/lvm/V_${vgname} - these lock files are opened, then flock()ed, and eventually either unlocked and later locked again, or potentially just unlink()ed with the lock held.

The unlink() can potentially cause the lock to desync and cause problems. Consider the following scenario with three processes (ordering is as is, the numbers are process numbers):

1.  open()
2.  open()
1.  flock() <-- succeeds
2.  flock() <-- blocks.
1.  unlink()
1.  close() <--@this point process 2's flock succeeds.
3.  open() <-- note that this ends up being a *different* file.
3.  flock() <-- succeeds.

At this point both 2 and 3 thinks they have the lock and that's wrong.

I actually saw an instance today where dmeventd had a file descriptor open to a deletect V_vggroup lockfile, so this *does* happen in the field. This also explains various lockups i've seen in the past, which I later figured out usually happened when dmeventd was running (So i put much effort into ensuring dmeventd never ever started up - which helped a lot).

Permitting I'm right the fix would be to fix _undo_flock in lib/locking/file_locking.c to not unlink the lockfile - ever. Or any other file that is used for locking purposes anywhere in the codebase for that matter.

Signature · 2 lines
Kind Regards,
Jaco Kroon
author
#1 lore raw

Hi All,

I've applied the attached patch to my installation, and the lock files are no longer unlinked. Since then a number of problems have gone away:

1. dmeventd (in spite of running a watch lvs in a terminal) no longer ends up with an open
2. I no longer seem to be getting lockups where I can't execute any lvm commands until I killall -9 dmeventd.
3. My snapshots no longer seem to go invalid because dmeventd doesn't extend them (during testing it still happened but I managed to figure out it's because my threshold for extending was set at 80%, and my initial LV was 12M, with 1M writes every 2 seconds to the original partition, and it would then go from <80% to 100% between checks, I since updated the threshold to 50% and checking at 15s intervals, and decreased the writes to every 5s which solved it - in production however I start with 1G snapshots, threshold@80 % and increase of 20%).

The only effective change of the patch is that it removes the code that unlinks the lockfile. To the best of my knowledge no safe way exists to unlink the lockfile without introducing lock race conditions, as such, the empty lock file has to remain on disk.

For your consideration,

Kind Regards,
Jaco

Quoted 29 lines · Jaco Kroon

Hi All,

LVM2 uses a locking scheme, relying on flock to maintain lock files for volume groups, by default /var/lock/lvm/V_${vgname} - these lock files are opened, then flock()ed, and eventually either unlocked and later locked again, or potentially just unlink()ed with the lock held.

The unlink() can potentially cause the lock to desync and cause problems. Consider the following scenario with three processes (ordering is as is, the numbers are process numbers):

1.  open()
2.  open()
1.  flock() <-- succeeds
2.  flock() <-- blocks.
1.  unlink()
1.  close() <--@this point process 2's flock succeeds.
3.  open() <-- note that this ends up being a *different* file.
3.  flock() <-- succeeds.

At this point both 2 and 3 thinks they have the lock and that's wrong.

I actually saw an instance today where dmeventd had a file descriptor open to a deletect V_vggroup lockfile, so this *does* happen in the field. This also explains various lockups i've seen in the past, which I later figured out usually happened when dmeventd was running (So i put much effort into ensuring dmeventd never ever started up - which helped a lot).

Quoted 4 lines

Permitting I'm right the fix would be to fix _undo_flock in lib/locking/file_locking.c to not unlink the lockfile - ever. Or any other file that is used for locking purposes anywhere in the codebase for that matter.

lib/locking/file_locking.c renamed from lib/locking/file_locking.c.o +0 -10
@@ -46,16 +46,6 @@
 
 static void _undo_flock(const char *file, int fd)
 {
-	struct stat buf1, buf2;
-
-	log_debug("_undo_flock %s", file);
-	if (!flock(fd, LOCK_NB | LOCK_EX) &&
-	    !stat(file, &buf1) &&
-	    !fstat(fd, &buf2) &&
-	    is_same_inode(buf1, buf2))
-		if (unlink(file))
-			log_sys_error("unlink", file);
-
 	if (close(fd) < 0)
 		log_sys_error("close", file);
 }

begin:vcard
fn:Jaco Kroon
n:Kroon;Jaco
org:Ultimate Linux Solutions CC
email;internet:[email protected]
title:Managing Member
tel;work:0873513298
tel;fax:0866488561
tel;cell:0845158255
url:http://www.uls.co.za/
version:2.1
end:vcard