Bonnie++ Burps on XFS
I've been using bonnie++ off and on for a long time. Suddenly, it has started failing when run against an XFS filesystem situated on a SATA drive. Here's the output of a run:
$ bench/sbin/bonnie++ -m "Luther 2.6.17-rc1 XFS SATA" \
Review
Quoted 3 lines
(quoted diff) +0
-3-d /archive/bench \ -s 3008 \ -n 1024 >> bonnie.out 2>> bonnie.err
Version: 1.03
Writing with putc()...done
Writing intelligently...done
Rewriting...done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...Bonnie: drastic I/O error (rmdir):
Directory not empty
Cleaning up test directory after error.
This is on a 2.6.17-rc1 Linux kernel. The interesting thing is that not even root cannot delete the directory in which bonnie++ is working:
[~]$ ls -ld /archive/bench/Bonnie.8426/
drwx------ 2 kwall users 4096 2006-04-05 09:06 /archive/bench/Bonnie.8426/ [~]$ ls -al /archive/bench/Bonnie.8426/
total 24
drwx------ 2 kwall users 4096 2006-04-05 09:06 ./
drwxr-xr-x 4 kwall users 42 2006-04-05 06:24 ../
[~]$ sudo rm -rf /archive/bench/Bonnie.8426
rm: cannot remove directory `/archive/bench/Bonnie.8426': Directory not empty
As you can see, the directory is empty, but a "rm" fails (as does "rmdir"). I'm retesting with a 2.6.16 kernel as I type. It isn't clear if the problems is XFS, some other piece of the kernel, or bonnie++. Neither dmesg nor syslog shows anything amiss. I suppose I could strace the run, but I'm not especially eager to deal with a multi-gigabyte output file if there is a more focused method to isolate the problem.
I've also written to bonnie++'s author.
Thanks for the help.
Best regards,
Kurt
Signature · 2 lines
The full impact of parenthood doesn't hit you until you multiply the number of your kids by 32 teeth.
Quoted 3 lines · Kurt Wall
I've been using bonnie++ off and on for a long time. Suddenly, it has started failing when run against an XFS filesystem situated on a SATA drive. Here's the output of a run:
[ Please report these things to [email protected]... ]
Quoted 1 line
Delete files in sequential order...Bonnie: drastic I/O error (rmdir):
Anything in your system log?
Quoted 5 lines
clear if the problems is XFS, some other piece of the kernel, or bonnie++. Neither dmesg nor syslog shows anything amiss. I suppose I could strace the run, but I'm not especially eager to deal with a multi-gigabyte output file if there is a more focused method to isolate the problem.
See if 2.6.16.1 fails too for starters then we'll take it from there.
cheers.
Signature · 1 line
Nathan
Quoted 10 lines · Nathan Scott
Quoted 3 lines · Kurt Wall
I've been using bonnie++ off and on for a long time. Suddenly, it has started failing when run against an XFS filesystem situated on a SATA drive. Here's the output of a run:
[ Please report these things to [email protected]... ]
Quoted 1 line
Delete files in sequential order...Bonnie: drastic I/O error (rmdir):
Anything in your system log?
Lemme answer that for you - "no". I've reproduced the problem, I'll get back to you once I've nutted out whats gone wrong.
Thanks for reporting it.
cheers.
Signature · 1 line
Nathan
Quoted 22 lines · Nathan Scott
Quoted 8 lines · Nathan Scott
Quoted 5 lines · Kurt Wall
I've been using bonnie++ off and on for a long time. Suddenly, it has started failing when run against an XFS filesystem situated on a SATA drive. Here's the output of a run:
Quoted 10 lines
[ Please report these things to [email protected]... ]
Quoted 3 lines
Delete files in sequential order...Bonnie: drastic I/O error (rmdir):
Anything in your system log?
Quoted 9 lines
Lemme answer that for you - "no". I've reproduced the problem, I'll get back to you once I've nutted out whats gone wrong.
Thanks for reporting it.
cheers.
Me too.
2.6.16
I had filesystem corruption and needed ran xfs_repair.
After I finished I removed most of the lost and found bits but was left with:
haze:/scratch/lost+found# ll total 28 drwxr-xr-x 4 root root 38 2006-04-02 08:29 ./ drwxrwxrwx 26 root root 4096 2006-04-04 18:25 ../ drwxrwxr-x 2 1046 1046 8192 2006-04-02 08:30 658616481/ drwxrwxr-x 2 1046 1046 8192 2006-04-02 08:22 823441168/ haze:/scratch/lost+found# rmdir * rmdir: `658616481': Directory not empty rmdir: `823441168': Directory not empty haze:/scratch/lost+found# ll * 658616481: total 12 drwxrwxr-x 2 1046 1046 8192 2006-04-02 08:30 ./ drwxr-xr-x 4 root root 38 2006-04-02 08:29 ../
823441168:
total 12
drwxrwxr-x 2 1046 1046 8192 2006-04-02 08:22 ./
drwxr-xr-x 4 root root 38 2006-04-02 08:29 ../
Obviously I had an fs corruption (due to raid failure due to some bogus libata errors) so this may not be the same thing.
David