Skip to contentSkip to conversation

Hang running fs_mark

DISCUSSION Josef Bacik 3 participants 14 messages Sep 25, 2008 – Sep 29, 2008 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hello,

Reporting this on behalf of ric. He was running the following fs_mark command

./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F

Seems it hung and wasn't making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we've just locked up behind a page lock somewhere. I have to run off to class so I can't look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks,

Josef

#1 lore raw
Quoted 4 lines · Josef Bacik

Hello,

Reporting this on behalf of ric. He was running the following fs_mark command

Quoted 8 lines

./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F

Seems it hung and wasn't making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we've just locked up behind a page lock somewhere. I have to run off to class so I can't look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks,

Which kernel was this?

-chris

#2 lore raw
Quoted 7 lines · Chris Mason
Quoted 6 lines · Josef Bacik

Hello,

Reporting this on behalf of ric. He was running the following fs_mark command

./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F

Quoted 10 lines
Quoted 6 lines

Seems it hung and wasn't making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we've just locked up behind a page lock somewhere. I have to run off to class so I can't look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks,

Which kernel was this?

Its 2.6.25.14-108.fc9.x86_64 and its kernel-unstable since yesterday I think he said, its just shy whatever patches you've pushed into git recently, it has yan's backref work plus a few other fixes. Thanks,

Josef

Quoted 1 line
#3 lore raw
Quoted 15 lines · Josef Bacik
Quoted 14 lines · Chris Mason
Quoted 12 lines · Josef Bacik

Hello,

Reporting this on behalf of ric. He was running the following fs_mark command

./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F

Seems it hung and wasn't making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we've just locked up behind a page lock somewhere. I have to run off to class so I can't look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks,

Quoted 6 lines
Quoted 1 line

Which kernel was this?

Its 2.6.25.14-108.fc9.x86_64 and its kernel-unstable since yesterday I think he said, its just shy whatever patches you've pushed into git recently, it has yan's backref work plus a few other fixes. Thanks,

Well, the trace is strange because it looks like everyone is waiting on IO. Any chance once of the procs was spinning in system time?

-chris

#4 lore raw
Quoted 29 lines · Chris Mason
Quoted 17 lines · Josef Bacik
Quoted 15 lines · Chris Mason
Quoted 13 lines · Josef Bacik

Hello,

Reporting this on behalf of ric. He was running the following fs_mark command

./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F

Seems it hung and wasn't making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we've just locked up behind a page lock somewhere. I have to run off to class so I can't look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks,

Quoted 9 lines
Quoted 2 lines

Which kernel was this?

Its 2.6.25.14-108.fc9.x86_64 and its kernel-unstable since yesterday I think he said, its just shy whatever patches you've pushed into git recently, it has yan's backref work plus a few other fixes. Thanks,

Quoted 6 lines

Well, the trace is strange because it looks like everyone is waiting on IO. Any chance once of the procs was spinning in system time?

-chris

It actually cleaned up very nicely after I killed the fs_mark processes & unmounted the btrfs file system. Before doing that, the box was sluggish and had the feeling of a system with something that might have been spinning (but that is just an observation, not measured in any strict sense).

ric

#5 lore raw
Quoted 20 lines · Ric Wheeler
Quoted 19 lines · Chris Mason
Quoted 17 lines · Josef Bacik
Quoted 15 lines · Chris Mason
Quoted 13 lines · Josef Bacik

Hello,

Reporting this on behalf of ric. He was running the following fs_mark command

./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F

Seems it hung and wasn't making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we've just locked up behind a page lock somewhere. I have to run off to class so I can't look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks,

Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

Which kernel was this?

It actually cleaned up very nicely after I killed the fs_mark processes & unmounted the btrfs file system. Before doing that, the box was sluggish and had the feeling of a system with something that might have been spinning (but that is just an observation, not measured in any strict sense).

Ok, I have that fs_mark test running here. How far did yours get before it stopped?

-chris

#6 lore raw
Quoted 31 lines · Chris Mason
Quoted 27 lines · Ric Wheeler
Quoted 25 lines · Chris Mason
Quoted 22 lines · Josef Bacik
Quoted 19 lines · Chris Mason
Quoted 4 lines · Josef Bacik

Hello,

Reporting this on behalf of ric. He was running the following fs_mark command

Quoted 10 lines

./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F

Seems it hung and wasn't making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we've just locked up behind a page lock somewhere. I have to run off to class so I can't look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks,

Which kernel was this?

Quoted 9 lines
Quoted 2 lines

It actually cleaned up very nicely after I killed the fs_mark processes & unmounted the btrfs file system. Before doing that, the box was sluggish and had the feeling of a system with something that might have been spinning (but that is just an observation, not measured in any strict sense).

Ok, I have that fs_mark test running here. How far did yours get before it stopped?

-chris

I had gone (in heavy fsync mode) up to about 8 million files on a 1TB s-ata disk:

17 8064000 20480 5.6 15301404

This is the new (no system sync() call) Chris special fs_mark. The rate had been quite reasonable, starting out at around 160 20k files/sec, went under 100 files/sec at around 3 million files and then fell under 50 files/sec at around 7.5 million before hitting this really low speed at just under 8 million.

Maybe it really was not hung, just extremely slow...

ric

#7 lore raw
Quoted 11 lines · Ric Wheeler
Quoted 6 lines

Ok, I have that fs_mark test running here. How far did yours get before it stopped?

-chris

I had gone (in heavy fsync mode) up to about 8 million files on a 1TB s-ata disk:

    17      8064000        20480          5.6         15301404
Quoted 8 lines

This is the new (no system sync() call) Chris special fs_mark. The rate had been quite reasonable, starting out at around 160 20k files/sec, went under 100 files/sec at around 3 million files and then fell under 50 files/sec at around 7.5 million before hitting this really low speed at just under 8 million.

Maybe it really was not hung, just extremely slow...

I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I get 155 files/sec ;) My hope is that we're spinning around due to bad accounting on the reserved extents, and that Yan's latest patch set will fix it.

-chris

#8 lore raw
Quoted 20 lines · Chris Mason
Quoted 7 lines · Ric Wheeler
Quoted 7 lines

Ok, I have that fs_mark test running here. How far did yours get before it stopped?

-chris

Quoted 9 lines

I had gone (in heavy fsync mode) up to about 8 million files on a 1TB s-ata disk:

    17      8064000        20480          5.6         15301404

This is the new (no system sync() call) Chris special fs_mark. The rate had been quite reasonable, starting out at around 160 20k files/sec, went under 100 files/sec at around 3 million files and then fell under 50 files/sec at around 7.5 million before hitting this really low speed at just under 8 million.

Maybe it really was not hung, just extremely slow...

Quoted 6 lines

I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I get 155 files/sec ;) My hope is that we're spinning around due to bad accounting on the reserved extents, and that Yan's latest patch set will fix it.

-chris

I can update & restart my test as well. It is an odd box (8 CPUs, only 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in testing out edge conditions ;-)

ric

#9 lore raw
Quoted 10 lines · Ric Wheeler
Quoted 7 lines

I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I get 155 files/sec ;) My hope is that we're spinning around due to bad accounting on the reserved extents, and that Yan's latest patch set will fix it.

-chris

I can update & restart my test as well. It is an odd box (8 CPUs, only 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in testing out edge conditions ;-)

I'll push out Yan's patches tomorrow. My box here is at 17.5 million files and still going at 148 files/sec

-chris

#10 lore raw
Quoted 9 lines · Chris Mason
Quoted 7 lines · Ric Wheeler
Quoted 7 lines

I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I get 155 files/sec ;) My hope is that we're spinning around due to bad accounting on the reserved extents, and that Yan's latest patch set will fix it.

-chris

Quoted 10 lines
Quoted 2 lines

I can update & restart my test as well. It is an odd box (8 CPUs, only 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in testing out edge conditions ;-)

I'll push out Yan's patches tomorrow. My box here is at 17.5 million files and still going at 148 files/sec

-chris

Sounds like a plan, thanks!

Ric

#11 lore raw
Quoted 15 lines · Ric Wheeler
Quoted 14 lines · Chris Mason
Quoted 11 lines · Ric Wheeler
Quoted 7 lines

I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I get 155 files/sec ;) My hope is that we're spinning around due to bad accounting on the reserved extents, and that Yan's latest patch set will fix it.

-chris

I can update & restart my test as well. It is an odd box (8 CPUs, only 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in testing out edge conditions ;-)

Quoted 8 lines
Quoted 7 lines

I'll push out Yan's patches tomorrow. My box here is at 17.5 million files and still going at 148 files/sec

-chris

Sounds like a plan, thanks!

I declined a bit down to 60 files/sec, but overnight made it up to 58 or so million files without stalling. It is possible that my metadata threshold changes caused problems for you, which might explain why my 4GB of ram lasted longer than your 1GB.

I'll try to rework the thresholds.

-chris

#12 lore raw
In reply to Ric Wheeler
Quoted 15 lines · Ric Wheeler
Quoted 14 lines · Chris Mason
Quoted 11 lines · Ric Wheeler
Quoted 7 lines

I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I get 155 files/sec ;) My hope is that we're spinning around due to bad accounting on the reserved extents, and that Yan's latest patch set will fix it.

-chris

I can update & restart my test as well. It is an odd box (8 CPUs, only 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in testing out edge conditions ;-)

Quoted 4 lines
Quoted 3 lines

I'll push out Yan's patches tomorrow. My box here is at 17.5 million files and still going at 148 files/sec

Sounds like a plan, thanks!

I turned up a long standing balancing bug while testing Yan's latest code. It took forever to track down, but I've finally pushed out his patches.

My fs_mark run never did stall, but I'm retesting now with all of the latest changes. You'll need to pull the latest unstable btrfs-progs and kernel from git.

-chris

#13 lore raw
In reply to Josef Bacik
Quoted 22 lines · Josef Bacik
Quoted 8 lines · Chris Mason
Quoted 6 lines · Josef Bacik

Hello,

Reporting this on behalf of ric. He was running the following fs_mark command

./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F

Quoted 11 lines
Quoted 7 lines

Seems it hung and wasn't making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we've just locked up behind a page lock somewhere. I have to run off to class so I can't look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks,

Which kernel was this?

Quoted 6 lines

Its 2.6.25.14-108.fc9.x86_64 and its kernel-unstable since yesterday I think he said, its just shy whatever patches you've pushed into git recently, it has yan's backref work plus a few other fixes. Thanks,

Josef

I can fire off a new test with the latest from unstable if that is of interest,

ric