Hang running fs_mark
Hello,
Reporting this on behalf of ric. He was running the following fs_mark command
./fs_mark -d /mnt/test -s 20480 -D 64 -t 8 -F
Seems it hung and wasn't making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we've just locked up behind a page lock somewhere. I have to run off to class so I can't look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks,
Josef
Quoted 4 lines · Josef Bacik
Hello,
Reporting this on behalf of ric. He was running the following
fs_markcommand
Quoted 8 lines
./fs_mark -d
/mnt/test-s 20480 -D 64 -t 8 -FSeems it hung and wasn't making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we've just locked up behind a page lock somewhere. I have to run off to class so I can't look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks,
Which kernel was this?
-chris
Quoted 7 lines · Chris Mason
Quoted 6 lines · Josef Bacik
Hello,
Reporting this on behalf of ric. He was running the following
fs_markcommand./fs_mark -d
/mnt/test-s 20480 -D 64 -t 8 -F
Quoted 10 lines
Quoted 6 lines
Seems it hung and wasn't making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we've just locked up behind a page lock somewhere. I have to run off to class so I can't look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks,
Which kernel was this?
Its 2.6.25.14-108.fc9.x86_64 and its kernel-unstable since yesterday I think he said, its just shy whatever patches you've pushed into git recently, it has yan's backref work plus a few other fixes. Thanks,
Josef
Quoted 1 line
Quoted 15 lines · Josef Bacik
Quoted 14 lines · Chris Mason
Quoted 12 lines · Josef Bacik
Hello,
Reporting this on behalf of ric. He was running the following
fs_markcommand./fs_mark -d
/mnt/test-s 20480 -D 64 -t 8 -FSeems it hung and wasn't making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we've just locked up behind a page lock somewhere. I have to run off to class so I can't look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks,
Quoted 6 lines
Quoted 1 line
Which kernel was this?
Its 2.6.25.14-108.fc9.x86_64 and its kernel-unstable since yesterday I think he said, its just shy whatever patches you've pushed into git recently, it has yan's backref work plus a few other fixes. Thanks,
Well, the trace is strange because it looks like everyone is waiting on IO. Any chance once of the procs was spinning in system time?
-chris
Quoted 29 lines · Chris Mason
Quoted 17 lines · Josef Bacik
Quoted 15 lines · Chris Mason
Quoted 13 lines · Josef Bacik
Hello,
Reporting this on behalf of ric. He was running the following
fs_markcommand./fs_mark -d
/mnt/test-s 20480 -D 64 -t 8 -FSeems it hung and wasn't making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we've just locked up behind a page lock somewhere. I have to run off to class so I can't look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks,
Quoted 9 lines
Quoted 2 lines
Which kernel was this?
Its 2.6.25.14-108.fc9.x86_64 and its kernel-unstable since yesterday I think he said, its just shy whatever patches you've pushed into git recently, it has yan's backref work plus a few other fixes. Thanks,
Quoted 6 lines
Well, the trace is strange because it looks like everyone is waiting on IO. Any chance once of the procs was spinning in system time?
-chris
It actually cleaned up very nicely after I killed the fs_mark processes & unmounted the btrfs file system. Before doing that, the box was sluggish and had the feeling of a system with something that might have been spinning (but that is just an observation, not measured in any strict sense).
ric
Quoted 20 lines · Ric Wheeler
Quoted 19 lines · Chris Mason
Quoted 17 lines · Josef Bacik
Quoted 15 lines · Chris Mason
Quoted 13 lines · Josef Bacik
Hello,
Reporting this on behalf of ric. He was running the following
fs_markcommand./fs_mark -d
/mnt/test-s 20480 -D 64 -t 8 -FSeems it hung and wasn't making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we've just locked up behind a page lock somewhere. I have to run off to class so I can't look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks,
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Which kernel was this?
It actually cleaned up very nicely after I killed the
fs_markprocesses & unmounted the btrfs file system. Before doing that, the box was sluggish and had the feeling of a system with something that might have been spinning (but that is just an observation, not measured in any strict sense).
Ok, I have that fs_mark test running here. How far did yours get before it stopped?
-chris
Quoted 31 lines · Chris Mason
Quoted 27 lines · Ric Wheeler
Quoted 25 lines · Chris Mason
Quoted 22 lines · Josef Bacik
Quoted 19 lines · Chris Mason
Quoted 4 lines · Josef Bacik
Hello,
Reporting this on behalf of ric. He was running the following
fs_markcommandQuoted 10 lines
./fs_mark -d
/mnt/test-s 20480 -D 64 -t 8 -FSeems it hung and wasn't making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we've just locked up behind a page lock somewhere. I have to run off to class so I can't look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks,
Which kernel was this?
Quoted 9 lines
Quoted 2 lines
It actually cleaned up very nicely after I killed the
fs_markprocesses & unmounted the btrfs file system. Before doing that, the box was sluggish and had the feeling of a system with something that might have been spinning (but that is just an observation, not measured in any strict sense).Ok, I have that
fs_marktest running here. How far did yours get before it stopped?-chris
I had gone (in heavy fsync mode) up to about 8 million files on a 1TB s-ata disk:
17 8064000 20480 5.6 15301404
This is the new (no system sync() call) Chris special fs_mark. The rate had been quite reasonable, starting out at around 160 20k files/sec, went under 100 files/sec at around 3 million files and then fell under 50 files/sec at around 7.5 million before hitting this really low speed at just under 8 million.
Maybe it really was not hung, just extremely slow...
ric
Quoted 11 lines · Ric Wheeler
Quoted 6 lines
Ok, I have that
fs_marktest running here. How far did yours get before it stopped?-chris
I had gone (in heavy fsync mode) up to about 8 million files on a 1TB s-ata disk:
17 8064000 20480 5.6 15301404
Quoted 8 lines
This is the new (no system
sync()call) Chris specialfs_mark. The rate had been quite reasonable, starting out at around 160 20k files/sec, went under 100 files/sec at around 3 million files and then fell under 50 files/sec at around 7.5 million before hitting this really low speed at just under 8 million.Maybe it really was not hung, just extremely slow...
I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I get 155 files/sec ;) My hope is that we're spinning around due to bad accounting on the reserved extents, and that Yan's latest patch set will fix it.
-chris
Quoted 20 lines · Chris Mason
Quoted 7 lines · Ric Wheeler
Quoted 7 lines
Ok, I have that
fs_marktest running here. How far did yours get before it stopped?-chris
Quoted 9 lines
I had gone (in heavy fsync mode) up to about 8 million files on a 1TB s-ata disk:
17 8064000 20480 5.6 15301404This is the new (no system
sync()call) Chris specialfs_mark. The rate had been quite reasonable, starting out at around 160 20k files/sec, went under 100 files/sec at around 3 million files and then fell under 50 files/sec at around 7.5 million before hitting this really low speed at just under 8 million.Maybe it really was not hung, just extremely slow...
Quoted 6 lines
I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I get 155 files/sec ;) My hope is that we're spinning around due to bad accounting on the reserved extents, and that Yan's latest patch set will fix it.
-chris
I can update & restart my test as well. It is an odd box (8 CPUs, only 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in testing out edge conditions ;-)
ric
Quoted 10 lines · Ric Wheeler
Quoted 7 lines
I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I get 155 files/sec ;) My hope is that we're spinning around due to bad accounting on the reserved extents, and that Yan's latest patch set will fix it.
-chris
I can update & restart my test as well. It is an odd box (8 CPUs, only 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in testing out edge conditions ;-)
I'll push out Yan's patches tomorrow. My box here is at 17.5 million files and still going at 148 files/sec
-chris
Quoted 9 lines · Chris Mason
Quoted 7 lines · Ric Wheeler
Quoted 7 lines
I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I get 155 files/sec ;) My hope is that we're spinning around due to bad accounting on the reserved extents, and that Yan's latest patch set will fix it.
-chris
Quoted 10 lines
Quoted 2 lines
I can update & restart my test as well. It is an odd box (8 CPUs, only 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in testing out edge conditions ;-)
I'll push out Yan's patches tomorrow. My box here is at 17.5 million files and still going at 148 files/sec
-chris
Sounds like a plan, thanks!
Ric
Quoted 15 lines · Ric Wheeler
Quoted 14 lines · Chris Mason
Quoted 11 lines · Ric Wheeler
Quoted 7 lines
I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I get 155 files/sec ;) My hope is that we're spinning around due to bad accounting on the reserved extents, and that Yan's latest patch set will fix it.
-chris
I can update & restart my test as well. It is an odd box (8 CPUs, only 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in testing out edge conditions ;-)
Quoted 8 lines
Quoted 7 lines
I'll push out Yan's patches tomorrow. My box here is at 17.5 million files and still going at 148 files/sec
-chris
Sounds like a plan, thanks!
I declined a bit down to 60 files/sec, but overnight made it up to 58 or so million files without stalling. It is possible that my metadata threshold changes caused problems for you, which might explain why my 4GB of ram lasted longer than your 1GB.
I'll try to rework the thresholds.
-chris
Quoted 15 lines · Ric Wheeler
Quoted 14 lines · Chris Mason
Quoted 11 lines · Ric Wheeler
Quoted 7 lines
I'm at 6.9 million files so far on a 500GB disk, and not surprisingly, I get 155 files/sec ;) My hope is that we're spinning around due to bad accounting on the reserved extents, and that Yan's latest patch set will fix it.
-chris
I can update & restart my test as well. It is an odd box (8 CPUs, only 1GB of DRAM and a single large 1TB s-ata drive). Hopefully useful in testing out edge conditions ;-)
Quoted 4 lines
Quoted 3 lines
I'll push out Yan's patches tomorrow. My box here is at 17.5 million files and still going at 148 files/sec
Sounds like a plan, thanks!
I turned up a long standing balancing bug while testing Yan's latest code. It took forever to track down, but I've finally pushed out his patches.
My fs_mark run never did stall, but I'm retesting now with all of the latest changes. You'll need to pull the latest unstable btrfs-progs and kernel from git.
-chris
Quoted 22 lines · Josef Bacik
Quoted 8 lines · Chris Mason
Quoted 6 lines · Josef Bacik
Hello,
Reporting this on behalf of ric. He was running the following
fs_markcommand./fs_mark -d
/mnt/test-s 20480 -D 64 -t 8 -FQuoted 11 lines
Quoted 7 lines
Seems it hung and wasn't making any progress. He managed to get some sysrq-t, which is at http://people.redhat.com/jwhiter/fs-mark-hang.txt towards the bottom of the document. He could ctrl+c and unmount the fs so its not a hard hang. Looks like we've just locked up behind a page lock somewhere. I have to run off to class so I can't look into it too deeply so throwing this out there hoping somebody else figures it out :). Thanks,
Which kernel was this?
Quoted 6 lines
Its 2.6.25.14-108.fc9.x86_64 and its kernel-unstable since yesterday I think he said, its just shy whatever patches you've pushed into git recently, it has yan's backref work plus a few other fixes. Thanks,
Josef
I can fire off a new test with the latest from unstable if that is of interest,
ric