futex hang with rpm in 2.6.17.1-2174_FC5
I had a dead nfs server that was causing some programs to pause, in particular 'yum install foo' was paused. I kill -9'd the yum related processes.
I fixed up the nfs server and was able to un-mount the file system. I subsequently killed many backed up updatedb and similar processes.
Now, there are no rpm processes, but if I try 'rpm [anything]' it hangs trying to open a futex:
open("/var/lib/rpm/Packages", O_RDONLY|O_LARGEFILE) = 4fcntl64(4, F_SETFD, FD_CLOEXEC) = 0fstat64(4, {st_mode=S_IFREG|0644, st_size=41390080, ...}) = 0 futex(0xb7ba178c, FUTEX_WAIT, 1, NULL <unfinished ...>
Is there any way to figure out what is causing this futex-wait?
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 10 lines · Ben Greear
I had a dead nfs server that was causing some programs to pause, in particular 'yum install foo' was paused. I kill -9'd the yum related processes.
I fixed up the nfs server and was able to un-mount the file system. I subsequently killed many backed up updatedb and similar processes.
Now, there are no rpm processes, but if I try 'rpm [anything]' it hangs trying to open a futex:
Quoted 6 lines
open("/var/lib/rpm/Packages", O_RDONLY|O_LARGEFILE)= 4fcntl64(4, F_SETFD, FD_CLOEXEC)= 0fstat64(4, {st_mode=S_IFREG|0644, st_size=41390080, ...})= 0 futex(0xb7ba178c,FUTEX_WAIT, 1, NULL <unfinished ...>Is there any way to figure out what is causing this futex-wait?
The dead rpm you killed left behind locks in its databases. rm -f /var/lib/rpm/__db* and it should work again.
Dave
Signature · 1 line
http://www.codemonkey.org.uk
Quoted 16 lines · Dave Jones
Quoted 15 lines · Ben Greear
I had a dead nfs server that was causing some programs to pause, in particular 'yum install foo' was paused. I kill -9'd the yum related processes.
I fixed up the nfs server and was able to un-mount the file system. I subsequently killed many backed up updatedb and similar processes.
Now, there are no rpm processes, but if I try 'rpm [anything]' it hangs trying to open a futex:
open("/var/lib/rpm/Packages", O_RDONLY|O_LARGEFILE)= 4fcntl64(4, F_SETFD, FD_CLOEXEC)= 0fstat64(4, {st_mode=S_IFREG|0644, st_size=41390080, ...})= 0 futex(0xb7ba178c,FUTEX_WAIT, 1, NULL <unfinished ...>
Quoted 5 lines
Quoted 1 line
Is there any way to figure out what is causing this futex-wait?
The dead rpm you killed left behind locks in its databases. rm -f
/var/lib/rpm/__db*and it should work again.
I'll give that a try, but shouldn't these locks clean themselves up when the process is killed or shouldn't rpm notice the previous process is dead and clean it up itself?
Thanks,
Ben
Quoted 3 lines
Dave
Signature · 1 line
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 11 lines · Ben Greear
Quoted 8 lines · Dave Jones
Quoted 4 lines · Ben Greear
I had a dead nfs server that was causing some programs to pause, in particular 'yum install foo' was paused. I kill -9'd the yum related processes.
The dead rpm you killed left behind locks in its databases. rm -f
/var/lib/rpm/__db*and it should work again.I'll give that a try, but shouldn't these locks clean themselves up when the process is killed
If you kill -9'd the processes, what do you expect to do the clean up work ?
Quoted 2 lines
or shouldn't rpm notice the previous process is dead and clean it up itself?
Sounds sensible to me and you, but in the past sensible ideas and rpm maintainers haven't gone hand in hand.
Dave
Signature · 1 line
http://www.codemonkey.org.uk
Quoted 7 lines · Dave Jones
Quoted 6 lines · Ben Greear
Quoted 5 lines · Dave Jones
Quoted 4 lines · Ben Greear
I had a dead nfs server that was causing some programs to pause, in particular 'yum install foo' was paused. I kill -9'd the yum related processes.
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
The dead rpm you killed left behind locks in its databases. rm -f
/var/lib/rpm/__db*and it should work again.I'll give that a try, but shouldn't these locks clean themselves up when the process is killed
If you kill -9'd the processes, what do you expect to do the clean up work ?
Well, you can do tricks with file handles so that they are automatically closed/deleted when a process exits, even with kill -9. Since this lock is evidently something in the kernel (since the kernel call is blocking), then it seems like a similar trick could be crafted.
Quoted 6 lines
Quoted 2 lines
or shouldn't rpm notice the previous process is dead and clean it up itself?
Sounds sensible to me and you, but in the past sensible ideas and rpm maintainers haven't gone hand in hand.
Ahhh :)
Thanks,
Ben
Signature · 1 line
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 3 lines · Ben Greear
Well, you can do tricks with file handles so that they are automatically closed/deleted when
a process exits, even with kill -9.
Nope, that cannot work. The lock object must be visible in the filesystem space.
The correct solution would be to use robust mutexes. But, as Dave said, it's easier said than implemented in rpm.
Quoted 7 lines · Ben Greear
Quoted 6 lines
Quoted 2 lines
or shouldn't rpm notice the previous process is dead and clean it up itself?
Sounds sensible to me and you, but in the past sensible ideas and rpm maintainers haven't gone hand in hand.
Ahhh :)
Well said. rpm's source tarball size doubles with each release and it contains such unexpected things as ELF manipulation library. I have no idea what business rpm can possibly have with parsing ELF headers.
--
vda