BUG_ON in I/O scheduler, bugme # 288
Jens, Andrew
The group here doing dbt2 workload measurements have hit a couple of problems APPARENTLY in the block I/O scheduler when doing write-intensive raw disk I/O through a DAC960 extremeraid 2000 controller. This wasn't a problem in 2.5.49. It has appeared since then.
I've filed a bug on the OSDL bugme database. You can read it at:
http://bugme.osdl.org/show_bug.cgi?id=288
I've also put a more complete report in my web site:
http://www.osdl.org/archive/dmo/deadline_bugon.
Begin with the README file.
For same reason, the README file isn't appearing on my web page. I'll look into that. In the mean time, I've included the contests of the README file below.
I'm about to try reproducing the problem on a smaller hardware configuration. Then, I'll test whether the same problem occurs with read intensive I/O.
Dave Olien
-------------README---------------------------------------------------
Summary:
BUG_ON and system hangs occuring while doing write-itensive RAW disk I/O to disks on a DAC960 extremeRAID 2000 controller.
It's possible the BUG_ON and system hangs are different, possibly even unrelated problems. But I'm grouping them together for now until I've had more time to investigate.
In this directory are:
DOT_CONFIG: the .config file for the kernel that was running.
disktest.tar: a tar source file for the disktest program,
disktest_2.5.59.sh: a script that runs the disktest program to
reproduce the system hanging events.
BUG_ON: A console listing of the BUG_ON event.
DISKTEST_STACKS: stack listings of the disktest threads that are
hung in I/O, taken by the sysrq stack trace command.
The kernel being run was 2.5.59. We also tried 2.5.59-mm2, and it failed as well.
This was NOT a problem on linux 2.5.49.
The distribution on that system was Redhat 7.3. So, the gcc compiler version was 2.96.
The hardware configuration originally used to produce the failure is:
8 Pentium III Xeon procssors.
16 gig of memory, but the kernel is configured to use only 4gig. DAC960 extreme raid 2000, with 2 scsi channels, 11 disks on
each chanel. Each disk is 70 gigabytes. Each disk is its own logical device.
The BUG_ON() was encountered running sapdb database with the dbt2 work load. The BUG_ON occurred at a time that the database was performing a checkpoint. This is a random write-intensive activity that is done over many disk devices.
The I/O is done on RAW devices.
Other times, the operating system didn't BUG_ON, but the system effectivly hung during these checkpoint episodes.
We discovered that disktest could reproduce the problem with disktest when run on the same hardware platform. I'm in the process of trying to reproduce the problem on a smaller configuration. I'll also see if it's only a write-intesive problem, or if there is a similar problem with read-intensive I/O.
Quoted 4 lines · Dave Olien
Jens, Andrew
Quoted 8 lines
The group here doing dbt2 workload measurements have hit a couple of problems APPARENTLY in the block I/O scheduler when doing write-intensive raw disk I/O through a DAC960 extremeraid 2000 controller. This wasn't a problem in 2.5.49. It has appeared since then.
I've filed a bug on the OSDL bugme database. You can read it at:
http://bugme.osdl.org/show_bug.cgi?id=288
The title is "2.5.59 and 2.5.50-mm2". I assume it should be 2.5.59-mm2??
Quoted 3 lines
I've also put a more complete report in my web site:
oooh, goody. A new stresstest tool.
Quoted 9 lines
Begin with the README file.
For same reason, the README file isn't appearing on my web page. I'll look into that. In the mean time, I've included the contests of the README file below.
I'm about to try reproducing the problem on a smaller hardware configuration. Then, I'll test whether the same problem occurs with read intensive I/O.
OK, thanks.
The important thing about direct-io is that it will frequently cause multiple I/Os to be in flight against the same disk sector. That will never happen with regular I/O because the pagecache acts as a synchronisation point.
Probably, this has tickled a bug in the I/O scheduler. Possibly in direct-io, too - that code's fairly fresh, and quite complex.
Yup, that should be 2.5.59-mm2. My typo.
Quoted 5 lines
Quoted 3 lines
I've filed a bug on the OSDL bugme database. You can read it at:
http://bugme.osdl.org/show_bug.cgi?id=288The title is "2.5.59 and 2.5.50-mm2". I assume it should be 2.5.59-mm2??
My test system's down right now. As soon as it comes up, I'll get onto reproducing it there.
Quoted 11 lines · Dave Olien
Yup, that should be 2.5.59-mm2. My typo.
Quoted 5 lines
Quoted 3 lines
I've filed a bug on the OSDL bugme database. You can read it at:
http://bugme.osdl.org/show_bug.cgi?id=288The title is "2.5.59 and 2.5.50-mm2". I assume it should be 2.5.59-mm2??
My test system's down right now. As soon as it comes up, I'll get onto reproducing it there.
I'm assuming vanilla 2.5.59, there's no BUG_ON() in -mm5 that line.
Signature · 1 line
Jens Axboe
Yup, the BUG_ON was in vanilla 2.5.59. The mm patch still shows the non-completion issue.
Quoted 11 lines · Jens Axboe
I'm assuming vanilla 2.5.59, there's no
BUG_ON()in -mm5 that line.Signature · 7 lines
Jens Axboe - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Quoted 17 lines · Dave Olien
Jens, Andrew
The group here doing dbt2 workload measurements have hit a couple of problems APPARENTLY in the block I/O scheduler when doing write-intensive raw disk I/O through a DAC960 extremeraid 2000 controller. This wasn't a problem in 2.5.49. It has appeared since then.
I've filed a bug on the OSDL bugme database. You can read it at:
http://bugme.osdl.org/show_bug.cgi?id=288I've also put a more complete report in my web site:
http://www.osdl.org/archive/dmo/deadline_bugon.
Begin with the README file.
Quoted 8 lines
For same reason, the README file isn't appearing on my web page. I'll look into that. In the mean time, I've included the contests of the README file below.
I'm about to try reproducing the problem on a smaller hardware configuration. Then, I'll test whether the same problem occurs with read intensive I/O.
Thanks for the report. Andrew, I think this may be because deadline_add_drq_rb puts "aliased" requests in the next_drq although they are not put on the sort or fifo lists. This is the problem I described to you before and exists in mm4.
Nick
Quoted 5 lines · Nick Piggin
I think this may be because
deadline_add_drq_rbputs "aliased" requests in thenext_drqalthough they are not put on the sort or fifo lists. This is the problem I described to you before and exists in mm4.
Yes, but 2.5.59 doesn't do that.
Quoted 10 lines · Andrew Morton
Quoted 5 lines · Nick Piggin
I think this may be because
deadline_add_drq_rbputs "aliased" requests in thenext_drqalthough they are not put on the sort or fifo lists. This is the problem I described to you before and exists in mm4.Yes, but 2.5.59 doesn't do that.
OK yeah, I thought he said 2.5.59 was OK.
Quoted 8 lines · Dave Olien
Jens, Andrew
The group here doing dbt2 workload measurements have hit a couple of problems APPARENTLY in the block I/O scheduler when doing write-intensive raw disk I/O through a DAC960 extremeraid 2000 controller. This wasn't a problem in 2.5.49. It has appeared since then.
Quoted 7 lines
I've filed a bug on the OSDL bugme database. You can read it at:
http://bugme.osdl.org/show_bug.cgi?id=288I've also put a more complete report in my web site:
A request got on the fifo, but not in the sort tree. This is most likely an alias. Ah yes I see it, it can happen when two requests are merged. I'll be back with a fix for this soon.
Signature · 1 line
Jens Axboe
OK, I was able to reproduce at least the problem with I/O apparently never completing on my smaller test machine. I've been assuming this was related to whatever was causing the BUG_ON(). But in case it isn't, I'm going to continue to look into what's going on on the running system and I'll let you know what I find.
In the mean time, once you've generated a patch, I'll give it a try as soon as I get it. I'll also pass it on to the dbt2 workload guys.
Quoted 8 lines · Jens Axboe
A request got on the fifo, but not in the sort tree. This is most likely an alias. Ah yes I see it, it can happen when two requests are merged. I'll be back with a fix for this soon.
Signature · 1 line
Jens Axboe