Skip to contentSkip to conversation

futex hang with rpm in 2.6.17.1-2174_FC5

DISCUSSION Ben Greear 4 participants 7 messages Oct 20, 2006 – Nov 27, 2006

lore
Share & tools
lore

Ben Greear

author
#0 lore raw

I had a dead nfs server that was causing some programs to pause, in particular 'yum install foo' was paused. I kill -9'd the yum related processes.

I fixed up the nfs server and was able to un-mount the file system. I subsequently killed many backed up updatedb and similar processes.

Now, there are no rpm processes, but if I try 'rpm [anything]' it hangs trying to open a futex:

open("/var/lib/rpm/Packages", O_RDONLY|O_LARGEFILE) = 4
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
fstat64(4, {st_mode=S_IFREG|0644, st_size=41390080, ...}) = 0 futex(0xb7ba178c, FUTEX_WAIT, 1, NULL <unfinished ...>

Is there any way to figure out what is causing this futex-wait?

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#1 lore raw
Quoted 10 lines · Ben Greear

I had a dead nfs server that was causing some programs to pause, in particular 'yum install foo' was paused. I kill -9'd the yum related processes.

I fixed up the nfs server and was able to un-mount the file system. I subsequently killed many backed up updatedb and similar processes.

Now, there are no rpm processes, but if I try 'rpm [anything]' it hangs trying to open a futex:

Quoted 6 lines

open("/var/lib/rpm/Packages", O_RDONLY|O_LARGEFILE) = 4
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
fstat64(4, {st_mode=S_IFREG|0644, st_size=41390080, ...}) = 0 futex(0xb7ba178c, FUTEX_WAIT, 1, NULL <unfinished ...>

Is there any way to figure out what is causing this futex-wait?

The dead rpm you killed left behind locks in its databases. rm -f /var/lib/rpm/__db* and it should work again.

Dave

Signature · 1 line
http://www.codemonkey.org.uk
author
#2 lore raw
Quoted 16 lines · Dave Jones
Quoted 15 lines · Ben Greear

I had a dead nfs server that was causing some programs to pause, in particular 'yum install foo' was paused. I kill -9'd the yum related processes.

I fixed up the nfs server and was able to un-mount the file system. I subsequently killed many backed up updatedb and similar processes.

Now, there are no rpm processes, but if I try 'rpm [anything]' it hangs trying to open a futex:

open("/var/lib/rpm/Packages", O_RDONLY|O_LARGEFILE) = 4
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
fstat64(4, {st_mode=S_IFREG|0644, st_size=41390080, ...}) = 0 futex(0xb7ba178c, FUTEX_WAIT, 1, NULL <unfinished ...>

Quoted 5 lines
Quoted 1 line

Is there any way to figure out what is causing this futex-wait?

The dead rpm you killed left behind locks in its databases. rm -f /var/lib/rpm/__db* and it should work again.

I'll give that a try, but shouldn't these locks clean themselves up when the process is killed or shouldn't rpm notice the previous process is dead and clean it up itself?

Thanks,
Ben

Quoted 3 lines

Dave

Signature · 1 line
Ben Greear <[email protected]> Candela Technologies Inc  http://www.candelatech.com
#3 lore raw
Quoted 11 lines · Ben Greear
Quoted 8 lines · Dave Jones
Quoted 4 lines · Ben Greear

I had a dead nfs server that was causing some programs to pause, in particular 'yum install foo' was paused. I kill -9'd the yum related processes.

The dead rpm you killed left behind locks in its databases. rm -f /var/lib/rpm/__db* and it should work again.

I'll give that a try, but shouldn't these locks clean themselves up when the process is killed

If you kill -9'd the processes, what do you expect to do the clean up work ?

Quoted 2 lines

or shouldn't rpm notice the previous process is dead and clean it up itself?

Sounds sensible to me and you, but in the past sensible ideas and rpm maintainers haven't gone hand in hand.

Dave

Signature · 1 line
http://www.codemonkey.org.uk
author
#4 lore raw
Quoted 7 lines · Dave Jones
Quoted 6 lines · Ben Greear
Quoted 5 lines · Dave Jones
Quoted 4 lines · Ben Greear

I had a dead nfs server that was causing some programs to pause, in particular 'yum install foo' was paused. I kill -9'd the yum related processes.

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

The dead rpm you killed left behind locks in its databases. rm -f /var/lib/rpm/__db* and it should work again.

I'll give that a try, but shouldn't these locks clean themselves up when the process is killed

If you kill -9'd the processes, what do you expect to do the clean up work ?

Well, you can do tricks with file handles so that they are automatically closed/deleted when a process exits, even with kill -9. Since this lock is evidently something in the kernel (since the kernel call is blocking), then it seems like a similar trick could be crafted.

Quoted 6 lines
Quoted 2 lines

or shouldn't rpm notice the previous process is dead and clean it up itself?

Sounds sensible to me and you, but in the past sensible ideas and rpm maintainers haven't gone hand in hand.

Ahhh :)

Thanks,
Ben

Signature · 1 line
Ben Greear <[email protected]> Candela Technologies Inc  http://www.candelatech.com
#5 lore raw
Quoted 3 lines · Ben Greear

Well, you can do tricks with file handles so that they are automatically closed/deleted when
a process exits, even with kill -9.

Nope, that cannot work. The lock object must be visible in the filesystem space.

The correct solution would be to use robust mutexes. But, as Dave said, it's easier said than implemented in rpm.

#6 lore raw
In reply to Ben Greear
Quoted 7 lines · Ben Greear
Quoted 6 lines
Quoted 2 lines

or shouldn't rpm notice the previous process is dead and clean it up itself?

Sounds sensible to me and you, but in the past sensible ideas and rpm maintainers haven't gone hand in hand.

Ahhh :)

Well said. rpm's source tarball size doubles with each release and it contains such unexpected things as ELF manipulation library. I have no idea what business rpm can possibly have with parsing ELF headers.
--
vda