Negative "ios_in_flight" in the 2.4 kernel
I've been looking at some "iostat" data from a 2.4.26 machine. The device utilizations are 100 percent, even when the disk is idle, which is mathematically impossible. By doing some digging, I discovered this is a kernel bug, caused by "hd->ios_in_flight" going negative. The relevant code appears to my untrained eyes to be in drivers/block/ll_rw_blk.c, specifically
static inline void down_ios(struct hd_struct *hd)
{
disk_round_stats(hd);
--hd->ios_in_flight;
}static inline void up_ios(struct hd_struct *hd)
{
disk_round_stats(hd);
++hd->ios_in_flight;
}Question: wouldn't a simple refusal to decrement ios_in_flight in "down_ios" if it's zero fix this, or am I missing something?
Ed Borasky
[email protected]
Quoted 14 lines · M. Edward Borasky
I've been looking at some "iostat" data from a 2.4.26 machine. The device utilizations are 100 percent, even when the disk is idle, which is mathematically impossible. By doing some digging, I discovered this is a kernel bug, caused by "
hd->ios_in_flight" going negative. The relevant code appears to my untrained eyes to be indrivers/block/ll_rw_blk.c, specificallystatic inline void
down_ios(struct hd_struct *hd){ disk_round_stats(hd); --hd->ios_in_flight; }
Quoted 8 lines
static inline void
up_ios(struct hd_struct *hd){ disk_round_stats(hd); ++hd->ios_in_flight; }Question: wouldn't a simple refusal to decrement
ios_in_flightin "down_ios" if it's zero fix this, or am I missing something?
That would paper over the real bug, but it will work for you.
Signature · 1 line
Jens Axboe
Quoted 5 lines · Jens Axboe
Quoted 3 lines
Question: wouldn't a simple refusal to decrement
ios_in_flightin "down_ios" if it's zero fix this, or am I missing something?That would paper over the real bug, but it will work for you.
What is the "real bug", then? What will "work for me" is accurate disk usage tick counts. The intent of these statistics is something known as Operational Analysis of Queueing Networks.
The "requirement" is that the operations on each device be accurately counted, and the "wall clock" time spent *waiting* for requests and the time spent *servicing* requests be accurately accumulated for each device. The sector count is a bonus.
Quoted 1 line
From these raw counters, one can, and iostat does, compute throughput,
utilization, average service time, average wait time and average queue length. An excellent and highly readable reference for the math involved can be found at
http://www.cs.washington.edu/homes/lazowska/qsp/Images/Chap_03.pdf
That is the intent behind these counters, and what will "work for me" is a kernel that captures the raw counters correctly. If forcing ios_in_flight to be non-negative is done at the expense of losing or gaining ticks in the wait or service time accumulators, then it will not work for me.
Ed Borasky
http://www.borasky-research.net
Quoted 20 lines · M. Edward Borasky
Quoted 5 lines · Jens Axboe
Quoted 3 lines
Question: wouldn't a simple refusal to decrement
ios_in_flightin "down_ios" if it's zero fix this, or am I missing something?That would paper over the real bug, but it will work for you.
What is the "real bug", then? What will "work for me" is accurate disk usage tick counts. The intent of these statistics is something known as Operational Analysis of Queueing Networks.
The "requirement" is that the operations on each device be accurately counted, and the "wall clock" time spent *waiting* for requests and the time spent *servicing* requests be accurately accumulated for each device. The sector count is a bonus.
Quoted 1 line
From these raw counters, one can, and iostat does, compute throughput,
utilization, average service time, average wait time and average queue length. An excellent and highly readable reference for the math involved can be found at
Quoted 7 lines
http://www.cs.washington.edu/homes/lazowska/qsp/Images/Chap_03.pdf
That is the intent behind these counters, and what will "work for me" is a kernel that captures the raw counters correctly. If forcing
ios_in_flightto be non-negative is done at the expense of losing or gaining ticks in the wait or service time accumulators, then it will not work for me.
Well something is deaccounting uncorrectly (doh), probably the disk/partition accounting logic is doing wrong in some condition, Jens?
void req_merged_io(struct request *req)
{
struct hd_struct *hd1, *hd2;
locate_hd_struct(req, &hd1, &hd2);
if (hd1)
down_ios(hd1);
if (hd2)
down_ios(hd2);
}void req_finished_io(struct request *req)
{
struct hd_struct *hd1, *hd2;
locate_hd_struct(req, &hd1, &hd2);
if (hd1)
account_io_end(hd1, req);
if (hd2)
account_io_end(hd2, req);
}We could eliminate that possibility if you ran your tests with a single non-partitioned disk, but thats just a guess.
Jens has more of a clue than I certainly.
Quoted 54 lines · Marcelo Tosatti
Quoted 20 lines · M. Edward Borasky
Quoted 5 lines · Jens Axboe
Quoted 3 lines
Question: wouldn't a simple refusal to decrement
ios_in_flightin "down_ios" if it's zero fix this, or am I missing something?That would paper over the real bug, but it will work for you.
What is the "real bug", then? What will "work for me" is accurate disk usage tick counts. The intent of these statistics is something known as Operational Analysis of Queueing Networks.
The "requirement" is that the operations on each device be accurately counted, and the "wall clock" time spent *waiting* for requests and the time spent *servicing* requests be accurately accumulated for each device. The sector count is a bonus.
Quoted 1 line
From these raw counters, one can, and iostat does, compute throughput,
utilization, average service time, average wait time and average queue length. An excellent and highly readable reference for the math involved can be found at
Quoted 7 lines
http://www.cs.washington.edu/homes/lazowska/qsp/Images/Chap_03.pdf
That is the intent behind these counters, and what will "work for me" is a kernel that captures the raw counters correctly. If forcing
ios_in_flightto be non-negative is done at the expense of losing or gaining ticks in the wait or service time accumulators, then it will not work for me.Well something is deaccounting uncorrectly (doh), probably the disk/partition accounting logic is doing wrong in some condition, Jens?
void
req_merged_io(struct request *req){ struct hd_struct *hd1, *hd2; locate_hd_struct(req, &hd1, &hd2); if (hd1) down_ios(hd1); if (hd2) down_ios(hd2); }void
req_finished_io(struct request *req){ struct hd_struct *hd1, *hd2; locate_hd_struct(req, &hd1, &hd2); if (hd1) account_io_end(hd1, req); if (hd2) account_io_end(hd2, req); }
Quoted 2 lines
We could eliminate that possibility if you ran your tests with a single non-partitioned disk, but thats just a guess.
It would be nice to know if this was a vanilla kernel or patched in some way. The only recent bug in this area I remember was a bad merge in the SUSE tree with the io_request_lock scaling patch.
(and don't trim the cc list when replying, at least not if you want people to see your message)
Signature · 1 line
Jens Axboe
Quoted 6 lines · Jens Axboe
Quoted 2 lines
We could eliminate that possibility if you ran your tests with a single non-partitioned disk, but thats just a guess.
It would be nice to know if this was a vanilla kernel or patched in some way. The only recent bug in this area I remember was a bad merge in the SUSE tree with the
io_request_lockscaling patch.
I have seen this with Red Hat 2.4.18 (from RH 8.0) kernels, Gentoo 2.4.25 and 2.4.26 kernels, on both single-disk and two-disk systems. Now that I think of it, I've seen this on both single-processor and multi-processor systems and with both SCSI and IDE drives. I have also seen these systems run for quite a while without ios_in_flight going negative. And I've never seen ios_in_flight lower than -1 or higher than 0 on an idle system. So my conclusion is that an extra downcount is fairly rare.
I saw a very similar bug listed in the LKML about a year ago. For example, see
http://search.luky.org/linux-kernel.2004/msg00025.html
and
http://search.luky.org/linux-kernel.2004/msg03278.html
I think I'll try rebooting the two-disk box (which is easier to get one truly idle disk on) and running bonnie++ periodically to see if I can get steady-state ios_in_flight values other than -1 and 0 on an idle system (between bonnie++ runs). I can set up "oprofile" on the Gentoo boxes if that will help.
One other note: all of these systems when "idle" have a small amount of write activity going on. The Red Hat boxes are using ext3 filesystems and the Gentoo systems are using reiserfs. Is constant low-level writing to be expected with journaling?